Acoustic processing method, acoustic processing system and program

Abstract

To generate target sound including a modulation component which reflects intention of a user.SOLUTION: An acoustic processing system 100 sets N control values Cy_1 to Cy_N according to an instruction from a user, and processes acoustic feature quantity Q of a basic component of target sound and modulation data M corresponding to N control values Cy_1 to Cy_N so as to generate an acoustic signal Z of target sound, in which N modulation components corresponding to N control values Cy_1 to Cy_N are added to the basic component.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to techniques for acoustic processing. [Background technology]

[0002] Various techniques for generating characteristic sounds such as growls have been proposed. For example, Patent Document 1 discloses a technique for converting the voice quality of a voice represented by an audio signal by adding subharmonic components between local peaks in the frequency spectrum of the audio signal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4649888 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the technology of Patent Document 1, each subharmonic is generated using a coefficient table prepared in advance. Therefore, there is room for further improvement from the perspective of reflecting the user's intention in the synthesized voice. Note that while the above description focuses on the synthesis of voice, similar issues are expected in the synthesis of any sound, including, for example, the sound of a musical instrument. In consideration of the above circumstances, one aspect of the present disclosure aims to generate a target sound that includes modulation components that reflect the user's intention. [Means for solving the problem]

[0005] In order to solve the above problems, an acoustic processing method according to one aspect of the present disclosure sets a plurality of control values ​​in response to instructions from a user, and processes acoustic features of fundamental components of a target sound and modulation data corresponding to the plurality of control values, thereby generating an acoustic signal of the target sound in which a plurality of modulation components corresponding to the plurality of control values ​​are added to the fundamental components.

[0006] An acoustic processing method according to another aspect of the present disclosure processes acoustic features of a fundamental component of a target sound and modulation data corresponding to a plurality of control values ​​using a trained first generative model, thereby generating an acoustic signal of the target sound in which a plurality of modulation components corresponding to the plurality of control values ​​are added to the fundamental component.

[0007] An acoustic processing system according to one aspect of the present disclosure includes a control value setting unit that sets a plurality of control values ​​in response to instructions from a user, and a signal generation unit that processes acoustic features of fundamental components of a target sound and modulation data according to the plurality of control values ​​to generate an acoustic signal of the target sound in which a plurality of modulation components corresponding to the plurality of control values ​​are added to the fundamental components.

[0008] A program according to one aspect of the present disclosure causes a computer system to function as a control value setting unit that sets a plurality of control values ​​in response to instructions from a user, and a signal generation unit that processes acoustic features of fundamental components of a target sound and modulation data according to the plurality of control values ​​to generate an acoustic signal of the target sound in which a plurality of modulation components corresponding to the plurality of control values ​​are added to the fundamental components. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram of a sound processing system according to a first embodiment. [Figure 2] FIG. 2 is an explanatory diagram of a modulation component. [Figure 3] FIG. 1 is a block diagram illustrating an example of a functional configuration of a sound processing system. [Figure 4] FIG. 10 is a schematic diagram of a setting screen. [Figure 5] FIG. 2 is a block diagram of a modulation control unit. [Figure 6] 10 is a flowchart of a sound generation process. [Figure 7] FIG. 10 is an explanatory diagram of a training process. [Figure 8] 10 is a flowchart of a training process. [Figure 9] FIG. 10 is a block diagram illustrating an example of the functional configuration of a sound processing system according to a second embodiment. [Figure 10] FIG. 2 is an explanatory diagram of a modulated pulse train. [Figure 11] FIG. 10 is a schematic diagram of a setting screen in a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0010] A: First embodiment 1 is a block diagram illustrating the configuration of a sound processing system 100 according to a first embodiment. The sound processing system 100 is a computer system that generates an arbitrary target sound. The target sound is a sound that is to be generated by the sound processing system 100. The target sound in the first embodiment is, for example, a musical sound such as a singing sound produced by a singer or a performance sound produced by a musical instrument.

[0011] The sound processing system 100 includes a control device 11, a storage device 12, a display device 13, an operation device 14, and a sound emitting device 15. The sound processing system 100 is realized by an information terminal such as a smartphone, a tablet terminal, or a personal computer. The sound processing system 100 may be realized by a single device, or may be realized by multiple devices configured separately from each other.

[0012] The control device 11 is composed of one or more processors that control each element of the sound processing system 100. For example, the control device 11 is composed of one or more types of processors, such as a central processing unit (CPU), a sound processing unit (SPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). The control device 11 generates an acoustic signal Z that represents the waveform of a target sound. The acoustic signal Z is expressed as a time series of multiple samples that represent the intensity of the target sound.

[0013] The storage device 12 is one or more memories that store programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is configured with a known storage medium such as a magnetic storage medium or a semiconductor storage medium. The storage device 12 may be configured with a combination of multiple types of storage media. Furthermore, the storage device 12 may be a portable storage medium that is detachable from the sound processing system 100, or a storage medium to which the control device 11 can write or read data via a communication network (e.g., cloud storage).

[0014] The storage device 12 stores musical score data S representing a piece of music. The musical score data S is data representing the musical score of a piece of music. Specifically, the musical score data S specifies the pitch and sound duration for each of the multiple notes that make up the piece of music. When the target note is a singing note, the musical score data S specifies phoneme symbols (e.g., lyrics) for each note in addition to the pitch and sound duration. Note that the musical score data S may also specify information such as performance symbols that represent musical expressions.

[0015] The display device 13 displays an image under the control of the control device 11. The display device 13 is configured with a display panel such as a liquid crystal panel or an organic EL (Electroluminescence) panel. The operation device 14 is an input device that receives instructions from the user U. The operation device 14 is, for example, an operator operated by the user U, or a touch panel that detects contact by the user U. Note that the display device 13 or the operation device 14, which are separate from the sound processing system 100, may be connected to the sound processing system 100 by wire or wirelessly.

[0016] The sound emitting device 15 reproduces the target sound represented by the acoustic signal Z. The sound emitting device 15 is, for example, a speaker or a headphone. For convenience, a D / A converter that converts the acoustic signal Z from digital to analog and an amplifier that amplifies the acoustic signal Z are not shown in the figure. Furthermore, the sound emitting device 15, which is separate from the acoustic processing system 100, may be connected to the acoustic processing system 100 by wire or wirelessly.

[0017] The target sound in the first embodiment includes a fundamental component and N modulation components (N is a natural number equal to or greater than 1). The fundamental component is, for example, the harmonic component and non-harmonic component of the target sound. The harmonic component is a periodic acoustic component having harmonic frequencies (F0, 2F0, 3F0, ...) that are integer multiples of the fundamental frequency F0. Specifically, the harmonic component includes a fundamental component of the fundamental frequency F0 and multiple overtone components. On the other hand, the non-harmonic component is a non-periodic noise component that spans a wide range of frequency domain. The non-harmonic component contributes, for example, to the breathiness of the target sound. Note that the target sound does not necessarily include non-harmonic components.

[0018] Each of the N modulation components is a supplementary acoustic component added to the fundamental component. Specifically, the modulation components are sideband components (subharmonics) that occur between the harmonic components (fundamental and overtone components) of the fundamental component. Examples of modulation components include distortion components with characteristic sound quality, such as growls or creaks.

[0019] FIG. 2 is an explanatory diagram of each modulated component. Each modulated component is an acoustic component that depends on the fundamental frequency F0 of the fundamental component (harmonic component). Specifically, each modulated component is an acoustic component located within the interval between two adjacent harmonic frequencies (F0, 2F0, 3F0, ...) on the frequency axis. That is, a modulated component exists on the high-frequency side and the low-frequency side of each harmonic frequency. Specifically, a modulated component exists at a frequency that is spaced higher or lower from each harmonic frequency by a frequency (hereinafter referred to as a "modulation frequency E_n") that has a predetermined relationship with the fundamental frequency F0 (n is a natural number). The modulated frequency E_n is, for example, a frequency equivalent to 1 / (n+1) of the fundamental frequency F0 (E_n=F0 / (n+1)). Note that FIG. 2 conveniently illustrates a modulated component (n=1) spaced from each harmonic frequency by a modulation frequency F0 / 2 and a modulated component (n=2) spaced from each harmonic frequency by a modulation frequency F0 / 3.

[0020] 3 is a block diagram illustrating an example of the functional configuration of the acoustic processing system 100. The control device 11 executes a program stored in the storage device 12 to realize a plurality of functions (an information generation unit 20, a control value setting unit 30, a modulation control unit 40, and a signal generation unit 50) for generating an acoustic signal Z.

[0021] The information generating unit 20 generates control data X relating to the target sound. The control data X includes a fundamental frequency F0, an acoustic feature Q, and N control values ​​Cx_1 to Cx_N.

[0022] As described above, the fundamental frequency F0 is the frequency of the fundamental component of the target sound among multiple harmonic components included in the fundamental component. The acoustic feature Q is an acoustic feature expressed in the frequency domain for the fundamental component. For example, the acoustic feature Q is MFSC (Mel Frequency Spectral Coefficients) that represent the outline of the intensity spectrum of the fundamental components (harmonic components and non-harmonic components). Note that the acoustic feature Q may be various features such as MFCC (Mel-Frequency Cepstrum Coefficients), amplitude spectrum, power spectrum, or Mel spectrum of the target sound.

[0023] The control value Cx_n (n=1 to N) is a control parameter (modulation depth) for controlling the n-th modulation component of the N modulation components of the target sound. For example, the larger the control value Cx_n, the greater the intensity of the n-th modulation component in the target sound.

[0024] The first generation unit 21 generates the condition data D. Specifically, the first generation unit 21 generates the condition data D from the musical score data S. The condition data D is generated for each unit period. The condition data D is data that specifies the conditions of the target note. Specifically, information about the note that represents the target note is included in the condition data D. The information about the note that represents the target note includes, for example, the pitch or volume of the note, and information about the notes before and after the note. Furthermore, if the target note is a singing note, the condition data D includes the phoneme code of each note. As can be understood from the above explanation, the condition data D can also be said to be data that alternatively expresses the characteristics of the musical score represented by the musical score data S.

[0025] The first generating unit 21 may generate the condition data D by processing the score data S using a generative model such as a deep neural network (DNN). The generative model is a statistical estimation model that learns the relationship between the score data S and the condition data D through machine learning.

[0026] The second generation unit 22 generates control data X (F0, Q, Cx_1 to Cx_N) by processing the condition data D. The second generation unit 22 processes the condition data D using a trained generative model Gx.

[0027] As described above, the condition data D specifies the conditions of the target sound, and the control data X represents the acoustic characteristics of the target sound. Therefore, the condition data D and the control data X are mutually correlated. The generative model Gx is a statistical estimation model that has learned the relationship between the condition data D and the control data X through prior machine learning. In other words, the generative model Gx generates control data X that is statistically appropriate for the condition data D.

[0028] The generative model Gx is realized by a combination of a program that causes the control device 11 to execute a calculation to generate control data X from condition data D and a plurality of variables (e.g., bias and weight values) that are applied to the calculation. The numerical values ​​of each of the plurality of variables are set in advance by machine learning. The second generation unit 22 generates control data X (F0, Q, Cx_1 to Cx_N) by processing the condition data D using the trained generative model Gx. The control data X for each unit period is generated from the condition data D for that unit period. The generative model Gx is an example of a "second generative model."

[0029] The generative model Gx is configured, for example, by a deep neural network. For example, any type of deep neural network, such as a recurrent neural network (RNN) or a convolutional neural network (CNN), may be used as the generative model Gx. The generative model Gx may be configured by combining multiple types of deep neural networks. In addition, additional elements, such as a long short-term memory (LSTM) or attention, may be installed in the generative model Gx.

[0030] The control value setting unit 30 in Fig. 3 sets N control values ​​Cy_1 to Cy_N corresponding to different modulation components of the target sound. The control value setting unit 30 of the first embodiment sets each control value Cy_n in response to an instruction from the user U via the operation device 14. Specifically, the control value setting unit 30 sets the control value Cy_n by changing each control value Cx_n generated by the information generating unit 20 (second generating unit 22) in response to an instruction from the user U. That is, the control value Cx_n is changed to the control value Cy_n in response to an instruction from the user U.

[0031] Of the N control values ​​Cx_1 to Cx_N, the control value Cx_n for which no change is instructed is used as the control value Cy_n. That is, for the modulation components for which no change is instructed, of the N modulation components, the control value Cx_n is determined as the control value Cy_n. As described above, the control value Cx_n is generated for each unit period, and therefore the N control values ​​Cy_1 to Cy_N are also generated for each unit period.

[0032] FIG. 4 is a schematic diagram of a screen (hereinafter referred to as "setting screen 60") for a user U to change the control value Cy_n. The setting screen 60 includes a musical score area 61, a variable area 62, and an operation area 63. The musical score area 61 displays a musical score represented by musical score data S. Specifically, the musical score area 61 is a piano roll image in which musical note images 611 representing each note specified by the musical score data S are arranged within an area having a time axis (horizontal axis) and a pitch axis (vertical axis). The variable area 62 displays a time series of each of the N control values ​​Cy_1 to Cy_N. The time axis is common to the musical score area 61 and the variable area 62.

[0033] N operators 631_1 to 631_N corresponding to different modulation components are arranged in the operation area 63. Each operator 631_n is a virtual operator that allows the user U to instruct a change in the control value Cy_n. The user U can arbitrarily change the control value Cy_n by operating the operator 631_n using the operation device 14. Specifically, the user U changes the control value Cy_n by operating the operator 631_n while referring to the score displayed in the score area 61 and the temporal transition of each control value Cy_n displayed in the variable area 62.

[0034] Note that the N operators 631_1 to 631_N for changing each control value Cy_n may be installed as physical operators (physical operators) on the operation device 14. In a configuration in which the N operators 631_1 to 631_N are installed on the operation device 14, the N operators 631_1 to 631_ on the setting screen 60 may be displayed or may be omitted.

[0035] The modulation control unit 40 in Fig. 3 generates time signals (fundamental wave signal R and modulated wave signal W_n) used to generate modulation components. Fig. 5 is a block diagram of the modulation control unit 40. The modulation control unit 40 of the first embodiment includes a fundamental wave generating unit 41 and a modulated wave generating unit 42.

[0036] The fundamental wave generating unit 41 generates a fundamental wave signal R. The fundamental wave signal R is a periodic signal that fluctuates at the fundamental frequency F0 of the fundamental component of the target sound. Specifically, the fundamental wave generating unit 41 generates a sine wave of a predetermined amplitude (for example, an amplitude of 1) that fluctuates at the fundamental frequency F0 as the fundamental wave signal R. The fundamental wave signal R is expressed as a time series of a plurality of samples.

[0037] The modulated wave generating unit 42 generates N modulated wave signals W_1 to W_N corresponding to different modulated components. Each modulated wave signal W_n is a periodic signal that fluctuates at a modulation frequency E_n that has a predetermined relationship with the fundamental frequency F0 of the target sound. Specifically, the modulated wave signal W_n corresponding to the n-th modulated component is a sine wave of a predetermined amplitude (for example, an amplitude of 1) that fluctuates at the modulation frequency E_n. Each modulated wave signal W_n is expressed as a time series of a plurality of samples.

[0038] The signal generation unit 50 in Fig. 3 generates an audio signal Z of the target sound by processing control data Y. As described above, the audio signal Z is a signal of the target sound in which N modulation components corresponding to different control values ​​Cy_n are added to a fundamental component. The control data Y includes an audio feature quantity Q of the fundamental component of the target sound and modulation data M related to each modulation component of the target sound. In other words, the signal generation unit 50 generates the audio signal Z of the target sound by processing the audio feature quantity Q of the fundamental component of the target sound and the modulation data M related to each modulation component.

[0039] The modulation data M is data corresponding to N control values ​​Cy_1 to Cy_N. Specifically, the modulation data M in the first embodiment includes a fundamental wave signal R, N modulated wave signals W_1 to W_N, and N control values ​​Cy_1 to Cy_N. The modulation data M is generated for each unit period. Specifically, the modulation data M for each unit period includes a time series of samples of the fundamental wave signal R within the unit period, a time series of samples of each modulated wave signal W_n within the unit period, and N control values ​​Cy_1 to Cy_N corresponding to the unit period. The control data Y for each unit period includes an acoustic feature Q for the unit period and the modulation data M for the unit period.

[0040] The signal generation unit 50 processes the control data Y (i.e., the acoustic feature quantity Q and the modulation data M) to generate an acoustic signal Z. The signal generation unit 50 generates the acoustic signal Z using a trained generative model Gy.

[0041] As described above, the control data Y includes the acoustic feature quantity Q of the fundamental component of the target sound and the modulation data M related to each modulation component of the target sound, and the acoustic signal Z represents the waveform of the target sound including the fundamental component and each modulation component. Therefore, the control data Y and the acoustic signal Z are mutually correlated. The generative model Gy is a statistical estimation model (a so-called neural vocoder) that has learned the relationship between the control data Y and the acoustic signal Z through prior machine learning. In other words, the generative model Gy generates an acoustic signal Z that is statistically appropriate for the control data Y.

[0042] The generative model Gy is realized by a combination of a program that causes the control device 11 to execute a calculation to generate an acoustic signal Z from control data Y and a plurality of variables (e.g., bias and weighting values) that are applied to the calculation. The numerical values ​​of each of the plurality of variables are set in advance by machine learning. The signal generating unit 50 generates the acoustic signal Z by processing the control data Y using the trained generative model Gy. From the control data Y for each unit period, a time series of a plurality of samples of the acoustic signal Z within that unit period is generated. The acoustic signal Z is constructed by interconnecting the time series of a plurality of samples generated by the signal generating unit 50 for each unit period. Note that the generative model Gy is an example of a "first generative model."

[0043] The generative model Gy is composed of, for example, a deep neural network. For example, any type of deep neural network, such as a recurrent neural network or a convolutional neural network, is used as the generative model Gx. The generative model Gx may be composed of a combination of multiple types of deep neural networks. In addition, additional elements, such as long-term short-term memory or attention, may be installed in the generative model Gy.

[0044] 6 is a flowchart of a process (hereinafter referred to as "sound generation process") in which the control device 11 generates the sound signal Z. For example, the sound generation process is repeated in a unit period on the time axis.

[0045] When the sound generation process is started, the control device 11 (information generation unit 20) processes the musical score data S to generate control data X for the target sound (Sa1). Specifically, a first process (first generation unit 21) for generating condition data D from the musical score data S and a second process (second generation unit 22) for generating control data X by processing the condition data D using the generative model Gx are executed. The control device 11 (control value setting unit 30) sets N control values ​​Cy_1 to Cy_N in response to instructions from the user U via the operation device 14 (Sa2).

[0046] The control device 11 (modulation control unit 40) generates a fundamental wave signal R and N modulated wave signals W_1 to W_N according to the fundamental frequency F0 included in the control data X (Sa3). Through the above processing, control data Y including an acoustic feature Q and modulation data M is generated. The modulation data M includes the fundamental wave signal R, N modulated wave signals W_1 to W_N, and N control values ​​Cy_1 to Cy_N.

[0047] The control device 11 (signal generation unit 50) processes the control data Y (i.e., the acoustic feature quantity Q and modulation data M) using the generation model Gy to generate an acoustic signal Z (Sa4). The acoustic signal Z generated by the control device 11 is supplied to the sound emitting device 15, whereby the target sound is reproduced.

[0048] As described above, in the first embodiment, by processing the acoustic feature quantity Q of the fundamental component of the target sound and the modulation data M according to the N control values ​​Cy_1 to Cy_N set in response to an instruction from the user U, an acoustic signal Z of the target sound is generated in which N modulation components corresponding to different control values ​​Cy_n are added to the fundamental component. Therefore, it is possible to generate a target sound including N modulation components that reflect the intention of the user U. In other words, it is possible to realize a unique customer experience in which a target sound including modulation components with characteristics desired by the user U can be generated, for example.

[0049] In particular, in the first embodiment, modulation data M including a fundamental wave signal R, N modulated wave signals W_1 to W_N, and N control values ​​Cy_1 to Cy_N is processed by a generative model Gy. Therefore, a target sound can be generated in which the level of each modulated component is controlled to give a natural auditory impression.

[0050] 7 is an explanatory diagram of a process (hereinafter referred to as "training process") for establishing a generative model Gy according to the first embodiment through machine learning. By executing a program stored in the storage device 12, the control device 11 functions as a training processing unit 80 in FIG. 7 in addition to the elements (information generating unit 20, control value setting unit 30, modulation control unit 40, and signal generating unit 50) illustrated in FIG. 3. The training processing unit 80 establishes a generative model Gy through machine learning using a plurality of training data T.

[0051] A plurality of training data T are stored in the storage device 12. Each training data T is teacher data in which training control data Yt and training audio signals Zt are associated with each other. The control data Yt includes an audio feature Q of a training reference sound and modulation data M related to each modulation component of the reference sound. The reference sound is sound (e.g., a singing sound or a performance sound) to which no modulation components have been added. The audio signal Zt of the training data T represents the waveform (a time series of multiple samples) of a target sound to be generated from the control data Yt of the training data T. The target sound represented by the audio signal Zt is a singing sound or a performance sound to which N modulation components have been added to the reference sound.

[0052] The modulation data M of the training data T includes a fundamental signal R, N modulated wave signals W_1 to W_N, and N control values ​​Cy_1 to Cy_N. The fundamental signal R of the training data T is a sine wave that fluctuates at the fundamental frequency F0 of the reference sound. Each modulated wave signal W_n of the training data T is a sine wave that fluctuates at a modulation frequency E_n. Each control value Cy_n of the training data T is a variable related to the n-th modulation component of the N modulation components to be added to the reference sound.

[0053] 8 is a flowchart of the training process. For example, the training process is started in response to an instruction from the user U via the operation device 14.

[0054] When the training process starts, the control device 11 (training processing unit 80) selects one of the multiple training data T (hereinafter referred to as "selected training data T") stored in the storage device 12 (Sb1). The control device 11 (training processing unit 80) iteratively updates multiple variables of an initial or provisional generative model Gy (hereinafter referred to as "provisional model Gy0") using the selected training data T (Sb2 to Sb4).

[0055] The control device 11 generates an acoustic signal Z by processing the control data Yt of the selected training data T using the provisional model Gy0 (Sb2). The control device 11 calculates a loss function that represents the error between the acoustic signal Z generated by the provisional model Gy0 and the acoustic signal Zt of the selected training data T (Sb3). The control device 11 updates multiple variables of the provisional model Gy0 so that the loss function is reduced (ideally minimized) (Sb4). For example, backpropagation is used to update each variable according to the loss function.

[0056] The control device 11 determines whether a predetermined termination condition is met (Sb5). The termination condition may be, for example, that the loss function falls below a predetermined threshold, or that the amount of change in the loss function falls below a predetermined threshold. If the termination condition is not met (Sb5: NO), the control device 11 selects unselected training data T stored in the storage device 12 as new selected training data T (Sb1). That is, the process of updating multiple variables of the provisional model Gy0 (Sb2 to Sb4) is repeated until the termination condition is met (Sb5: YES). If the termination condition is met (Sb5: YES), the control device 11 terminates the training process. The provisional model Gy0 at the time the termination condition is met is determined to be the trained generative model Gy.

[0057] As can be understood from the above explanation, the generative model Gy learns the latent relationship between the control data Yt and the acoustic signal Zt in multiple sets of training data T. Therefore, the trained generative model Gy outputs an acoustic signal Z that is statistically valid for unknown control data Y based on the above relationship.

[0058] As described above, in the first embodiment, a trained generative model Gy is used to generate an audio signal Z. Therefore, it is possible to generate an audio signal Z that reflects the trends present in the multiple pieces of training data T used to train the generative model Gy. Specifically, it is possible to generate a target sound to which auditorily natural modulation components are added.

[0059] Although the above description focuses on the training of the generative model Gy, the control device 11 (training processing unit 80) also trains the generative model Gx using the same procedure as the example shown in Fig. 8. A plurality of training data including training condition data D and training control data X is used to train the generative model Gx. Specifically, the control device 11 (training processing unit 80) updates a plurality of variables of the generative model Gx so as to reduce (ideally minimize) the error between the control data X generated by the initial or provisional generative model Gx from the condition data D of each training data and the learning control data X included in the training data.

[0060] As described above, in the first embodiment, the acoustic feature Q and fundamental frequency F0 of the fundamental component of the target sound, and N control values ​​Cx_1 to Cx_N corresponding to different modulation components are generated by the trained generative model Gx. Therefore, the aforementioned effect of being able to generate a target sound containing a plurality of perceptually natural modulation components is particularly effective.

[0061] B: Second embodiment A second embodiment will be described. Note that, for elements in the following exemplary aspects that have the same functions as those in the first embodiment, the same reference numerals as those in the first embodiment will be used, and detailed descriptions of each will be omitted as appropriate.

[0062] 9 is a block diagram illustrating the functional configuration of the sound processing system 100 according to the second embodiment. In the second embodiment, the configuration and operation of the modulation control unit 40 differ from those in the first embodiment.

[0063] The modulation control unit 40 of the second embodiment generates a modulated pulse train P. Specifically, the modulation control unit 40 generates the modulated pulse train P in accordance with the fundamental frequency F0 of the target sound generated by the information generation unit 20 and N control values ​​Cy_1 to Cy_N set by the control value setting unit 30. Similar to the first embodiment, the control value setting unit 30 sets the N control values ​​Cy_1 to Cy_N in accordance with an instruction from the user U.

[0064] FIG. 10 is an explanatory diagram of a modulated pulse train P. The modulated pulse train P is a time series of multiple pulses p. The modulated pulse train P is expressed as a time series of multiple samples on the time axis. The multiple pulses p of the modulated pulse train P are arranged on the time axis at a period corresponding to the fundamental frequency F0 (hereinafter referred to as "fundamental period B"). The fundamental period B is the reciprocal of the fundamental frequency F0 (B=1 / F0).

[0065] The levels of the multiple pulses p that make up the modulated pulse train P fluctuate at a period corresponding to each modulation frequency E_n. For example, FIG. 10 illustrates a case where the levels fluctuate at a period (1 / E_3) corresponding to the modulation frequency E_3. As described above, the modulation frequency E_n is 1 / (n+1) of the fundamental frequency F0, so the period corresponding to the modulation frequency E_3 has a time length that is (n+1) times the fundamental period B. In other words, as illustrated in FIG. 10, the levels of the multiple pulses p fluctuate repeatedly at a period that is four times the fundamental period B.

[0066] Furthermore, the levels of the multiple pulses p that make up the modulated pulse train P vary according to N control values ​​Cy_1 to Cy_N. That is, the fluctuation range δ in the period corresponding to the modulation frequency E_n in the modulated pulse train P is controlled according to the control value Cy_n. For example, the larger the control value Cy_n, the larger the fluctuation range δ in the period corresponding to the modulation frequency E_n. That is, the larger the control value Cy_n, the more emphasized the fluctuation in the period corresponding to the modulation frequency E_n in the modulated pulse train P (i.e., the nth modulation component).

[0067] 9, the control data Y of the second embodiment includes an acoustic feature quantity Q of the fundamental component of the target sound and modulation data M related to each modulation component of the target sound. The modulation data M of the second embodiment includes a modulation pulse train P generated by the modulation control unit 40. Specifically, the modulation data M of each unit period includes a time series of samples of the modulation pulse train P within the unit period.

[0068] As in the first embodiment, the signal generation unit 50 processes the control data Y using the generation model Gy to generate the audio signal Z. As described above, the control data Y includes the audio feature quantity Q of the fundamental component of the target sound and modulation data M related to each modulation component of the target sound, and the audio signal Z represents the waveform of the target sound including the fundamental component and each modulation component. Therefore, the control data Y and the audio signal Z are mutually correlated. The generation model Gy in the second embodiment is a statistical estimation model (a so-called neural vocoder) that has learned the relationship between the control data Y and the audio signal Z through prior machine learning. In other words, as in the first embodiment, the generation model Gy generates an audio signal Z that is statistically appropriate for the control data Y. The procedure of the audio generation process is the same as in the first embodiment.

[0069] The second embodiment also achieves the same effects as the first embodiment. Moreover, in the second embodiment, a modulated pulse train P in which the level of each of a plurality of pulses p varies in a period corresponding to a modulation frequency E_n in accordance with N control values ​​Cy_1 to Cy_N corresponding to different modulation components is processed by a generation model Gy. Therefore, it is possible to generate a target sound in which the level of each modulation component is controlled to give an auditory natural impression.

[0070] C: Modified Example Specific modified embodiments that can be added to each of the embodiments exemplified above are exemplified below. Two or more embodiments arbitrarily selected from the following examples may be combined as appropriate within the scope of not being mutually contradictory.

[0071] (1) The setting screen 60 displayed on the display device 13 is not limited to the image exemplified in Fig. 4. For example, the setting screen 60 exemplified in Fig. 11 may be displayed on the display device 13. The setting screen 60 in Fig. 11 is an image in which the musical score area 61 in Fig. 4 has been replaced with a waveform area. The waveform of the audio signal Z is displayed in the waveform area.

[0072] (2) In the above embodiments, the control data Y is generated for each unit period of a predetermined length, but the time unit of the control data Y is arbitrary. For example, the control data Y may be generated for each sample of the audio signal Z. The signal generating unit 50 generates one sample of the audio signal Z from the control data Y corresponding to one point on the time axis (hereinafter referred to as the "processing point").

[0073] The control data Y corresponding to each processing time point includes an acoustic feature Q corresponding to the processing time point. The acoustic feature Q corresponding to one processing time point is generated by interpolating the acoustic feature Q generated by the information generating unit 20 for each unit period.

[0074] In the first embodiment, the modulated data M corresponding to one processing time point includes one sample of the fundamental wave signal R corresponding to the processing time point and one sample of each modulated wave signal W_n corresponding to the processing time point. Also, the modulated data M corresponding to one processing time point includes N control values ​​Cy_1 to Cy_N corresponding to the processing time point. The control value Cy_n corresponding to one processing time point is generated by interpolating the control value Cx_n generated by the information generating unit 20 for each unit period. In the second embodiment, the modulated data M corresponding to one processing time point includes one sample of the modulated pulse train P corresponding to the processing time point.

[0075] (3) In each of the above embodiments, for convenience, the sound processing system 100 is provided with the training processing unit 80. However, the training processing unit 80 may be mounted on a system (machine learning system) separate from the sound processing system 100. The machine learning system is realized by a server device such as a web server, and constructs a generative model G (Gx, Gy) through the above-described training process. The generative model G constructed by the machine learning system is transmitted to the sound processing system 100.

[0076] (4) In the above-described embodiments, the control values ​​Cy_n are changed in response to instructions from the user U. However, the configuration in which the instructions from the user U are reflected in the control values ​​Cy_n may be omitted. For example, the control values ​​Cx_n generated by the information generating unit 20 may be included in the control data Y as definitive control values ​​Cy_n.

[0077] (5) In the above embodiments, the audio signal Z is generated using the musical score data S as input. However, the input data for generating the audio signal Z is not limited to the musical score data S.

[0078] For example, a possible configuration is one in which character string data representing a time sequence of multiple characters is input to generate an audio signal Z. The condition data D is, for example, data representing each phoneme corresponding to the character string. In the above configuration, an audio signal Z is generated with the target sound being the voice reading out a character string specified by the character string data.

[0079] Also, a configuration is envisioned in which the acoustic signal Z is generated using as input voice data representing the voice of an arbitrary speaker such as a user U. The condition data D is data representing the frequency characteristics of the voice. In the above configuration, for example, the acoustic signal Z representing a target sound obtained by converting the voice quality of the voice represented by the voice data is generated. In other words, the sound processing system 100 functions as a voice quality conversion device.

[0080] As can be understood from the above explanation, the content of the input data used to generate the audio signal Z is arbitrary, and various types of input data such as character string data or audio data in addition to the musical score data S are used to generate the audio signal Z. The condition data D is comprehensively expressed as data that specifies the conditions of the target sound.

[0081] (6) In the above-described embodiments, a deep neural network is used as the generative model G (Gx, Gy), but the configuration of the generative model G is not limited to the above examples. For example, a statistical model such as a hidden Markov model (HMM) or a support vector machine (SVM) can also be used as the generative model G.

[0082] (7) The sound processing system 100 may be realized by a server device that communicates with an information device such as a smartphone or a tablet terminal. For example, the sound processing system 100 generates an audio signal Z using musical score data S received from the information device and transmits the audio signal Z to the information device. An instruction to change each control value Cy_n is transmitted from the information device to the sound processing system 100.

[0083] In addition, in a configuration in which the condition data D is transmitted from the information device to the acoustic processing system 100, the first generation unit 21 may be omitted. In addition, in a configuration in which the control data X is transmitted from the information device to the acoustic processing system 100, the information generation unit 20 may be omitted.

[0084] (8) As described above, the functions of the acoustic processing system 100 illustrated above are realized by cooperation between one or more processors constituting the control device 11 and a program stored in the storage device 12. The program according to the present disclosure may be provided in a form stored on a computer-readable recording medium and installed on a computer. The recording medium may be, for example, a non-transitory recording medium, such as an optical recording medium (optical disk) such as a CD-ROM, but may also include any known type of recording medium, such as a semiconductor recording medium or a magnetic recording medium. Note that a non-transitory recording medium includes any recording medium other than a transient, propagating signal, and does not exclude volatile recording media. Furthermore, in a configuration in which a distribution device distributes a program via a communication network, the storage medium that stores the program in the distribution device corresponds to the non-transitory recording medium described above.

[0085] D: Notes From the above-described exemplary embodiments, the following configurations can be understood, for example.

[0086] An acoustic processing method according to one aspect (aspect 1) of the present disclosure sets a plurality of control values ​​in response to instructions from a user, and processes acoustic features of fundamental components of a target sound and modulation data in accordance with the plurality of control values ​​to generate an acoustic signal of the target sound in which a plurality of modulation components corresponding to the plurality of control values ​​are added to the fundamental components. According to the above aspect, by processing the acoustic features of the fundamental components of the target sound and modulation data in accordance with a plurality of control values ​​set in response to instructions from the user, an acoustic signal of the target sound in which a plurality of modulation components corresponding to different control values ​​are added to the fundamental components is generated. Therefore, a target sound including a plurality of modulation components that reflects the user's intention can be generated.

[0087] In a specific example (Aspect 2) of Aspect 1, the acoustic signal is generated by processing the acoustic features and the modulation data using a trained first generative model. In the above aspect, the trained first generative model is used to generate the acoustic signal. Therefore, it is possible to generate an acoustic signal that reflects trends present in the multiple training data used to train the first generative model. Specifically, it is possible to generate a target sound to which a perceptually natural modulation component is added.

[0088] In a specific example (Aspect 3) of Aspect 2, the modulation data includes a fundamental wave signal that is a sine wave having a fundamental frequency of the fundamental component, a plurality of modulated wave signals that are sine waves having different modulation frequencies that have a predetermined relationship with the fundamental frequency, and the plurality of control values. According to the above aspect, the modulation data that includes the fundamental wave signal, the modulated wave signal, and the plurality of control values ​​is processed by a first generative model. Therefore, it is possible to generate a target sound in which the level of each modulated component is controlled to give an auditory natural impression.

[0089] In a specific example (Aspect 4) of Aspect 2, the modulation data includes a modulated pulse train in which a plurality of pulses are arranged on a time axis at a period corresponding to the fundamental frequency of the fundamental component, and levels of the plurality of pulses vary at a period corresponding to each of the modulation frequencies in accordance with a plurality of control values ​​that respectively specify levels of the plurality of modulation components. According to the above aspect, a modulated pulse train in which the level of each of the plurality of pulses varies at a period corresponding to the modulation frequency in accordance with control values ​​corresponding to different modulation components is processed by a first generative model. Therefore, it is possible to generate a target sound in which the level of each modulation component is controlled to give an auditorily natural impression.

[0090] In a specific example (Aspect 5) of any of Aspects 2 to 4, condition data specifying the conditions of the target sound is further processed by a trained second generative model to generate acoustic features and fundamental frequencies of the fundamental components and the plurality of control values, and in setting the plurality of control values, the plurality of control values ​​generated by the second generative model are changed in accordance with instructions from the user. According to the above aspects, the acoustic features and fundamental frequencies of the fundamental components and the plurality of control values ​​corresponding to different modulation components are generated by the trained second generative model. Therefore, the aforementioned effect of being able to generate a target sound containing a plurality of modulation components that is auditorily natural is particularly effective.

[0091] However, the technology of Patent Document 1 has a problem in that the acoustic characteristics of the converted voice tend to be simple because a coefficient table prepared in advance is used for voice quality conversion. In other words, there is room for further improvement from the viewpoint of generating a target sound to which a perceptually natural modulation component is added.

[0092] In order to solve the above problems, an acoustic processing method according to one aspect (aspect 6) of the present disclosure processes acoustic features of fundamental components of a target sound and modulation data corresponding to a plurality of control values ​​using a trained first generative model, thereby generating an acoustic signal of the target sound in which a plurality of modulation components corresponding to the plurality of control values ​​are added to the fundamental components. According to the above aspect, since the acoustic features of the fundamental components of the target sound and modulation data corresponding to a plurality of control values ​​are processed by the first generative model, it is possible to generate a target sound including a plurality of modulation components that is auditorily natural.

[0093] An acoustic processing system according to one aspect (aspect 7) of the present disclosure includes a control value setting unit that sets a plurality of control values ​​in response to instructions from a user, and a signal generation unit that processes acoustic features of fundamental components of a target sound and modulation data according to the plurality of control values ​​to generate an acoustic signal of the target sound in which a plurality of modulation components corresponding to the plurality of control values ​​are added to the fundamental components.

[0094] In a specific example (Aspect 8) of Aspect 7, the control value setting unit sets the plurality of control values ​​in response to operations by the user on a plurality of physical operators. In another specific example (Aspect 9) of Aspect 7, a display device is provided that displays a plurality of operators, and the control value setting unit sets the plurality of control values ​​in response to operations by the user on the plurality of operators. In a specific example (Aspect 10) of Aspect 9, the display device displays the plurality of control values. According to Aspect 10, the user can adjust each control value by operating the operators while checking the numerical values ​​of each of the plurality of control values.

[0095] A program according to one aspect (aspect 11) of the present disclosure causes a computer system to function as a control value setting unit that sets multiple control values ​​in response to instructions from a user, and a signal generation unit that processes acoustic features of fundamental components of a target sound and modulation data according to the multiple control values ​​to generate an acoustic signal of the target sound in which multiple modulation components corresponding to the multiple control values ​​are added to the fundamental components. [Explanation of symbols]

[0096] 100...acoustic processing system, 11...control device, 12...storage device, 13...display device, 14...operation device, 15...sound emission device, 20...information generation unit, 21...first generation unit, 22...second generation unit, 30...control value setting unit, 40...modulation control unit, 41...fundamental wave generation unit, 42...modulated wave generation unit, 50...signal generation unit, 60...setting screen, 80...training processing unit.

Claims

1. A plurality of control values ​​are set according to instructions from a user, By processing the acoustic feature amount of the fundamental component of the target sound and the modulation data corresponding to the plurality of control values, an acoustic signal of the target sound is generated in which a plurality of modulation components corresponding to the plurality of control values ​​are added to the fundamental component. An audio processing method implemented by a computer system.

2. In generating the acoustic signal, the acoustic feature and the modulation data are processed by a trained first generative model to generate the acoustic signal. The acoustic processing method according to claim 1.

3. The modulation data is A fundamental signal that is a sine wave having a fundamental frequency of the fundamental component; a plurality of modulated wave signals which are sine waves having different modulated frequencies in a predetermined relationship to the fundamental frequency; and the plurality of control values. The acoustic processing method according to claim 2.

4. The modulation data is a modulated pulse train in which a plurality of pulses are arranged on a time axis at a period corresponding to the fundamental frequency of the fundamental component; The levels of the plurality of pulses vary in a period corresponding to each of the modulation frequencies in response to a plurality of control values ​​that respectively specify the levels of the plurality of modulation components. The acoustic processing method according to claim 2.

5. moreover, generating an acoustic feature quantity and a fundamental frequency of the basic component and the plurality of control values ​​by processing condition data specifying conditions of the target sound using a trained second generation model; In setting the plurality of control values, the plurality of control values ​​generated by the second generative model are changed in response to an instruction from the user. The sound processing method according to any one of claims 2 to 4.

6. By processing the acoustic feature quantity of the fundamental component of the target sound and the modulation data corresponding to the plurality of control values ​​by a trained first generation model, an acoustic signal of the target sound is generated in which a plurality of modulation components corresponding to the plurality of control values ​​are added to the fundamental component. An audio processing method implemented by a computer system.

7. a control value setting unit that sets a plurality of control values ​​in response to an instruction from a user; a signal generating unit that processes an acoustic feature amount of a fundamental component of a target sound and modulation data corresponding to the plurality of control values ​​to generate an acoustic signal of the target sound in which a plurality of modulation components corresponding to the plurality of control values ​​are added to the fundamental component; An audio processing system comprising:

8. The control value setting unit sets the control values ​​in response to operations by the user on a plurality of physical operators. The sound processing system of claim 7.

9. A display device is provided for displaying a plurality of operators, The control value setting unit sets the control values ​​in response to operations by the user on the plurality of operators. The sound processing system of claim 7.

10. The display device displays the plurality of control values. The sound processing system of claim 9.

11. a control value setting unit that sets a plurality of control values ​​in response to an instruction from a user; a signal generating unit that processes an acoustic feature amount of a fundamental component of the target sound and modulation data corresponding to the plurality of control values ​​to generate an acoustic signal of the target sound in which a plurality of modulation components corresponding to the plurality of control values ​​are added to the fundamental component; A program that causes a computer system to function as a

Images