Voice processing method, voice processing system, program and voice processing software

Abstract

To generate audibly natural reproduced voice in voice quality conversion of voice signals set in multiple tracks.SOLUTION: A voice processing system consists of: an allocation unit that allocates to each of a plurality of tracks in music production software a voice processing plug-in including voice quality conversion of voice signals set to the track; an adjustment unit 43 that differs a first music parameter related to a first voice signal after processing by the voice processing plug-in allocated to a first track among the plurality of tracks, and a second music parameter related to a second voice signal after processing by the voice processing plug-in allocated to a second track that is different from the first track.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to techniques for processing audio signals. [Background technology]

[0002] Various techniques for converting voice quality have been proposed in the past. For example, Patent Document 1 discloses a configuration in which intermediate feature data is generated by processing acoustic feature data generated from acoustic data with an acoustic encoder, and the intermediate feature data is then processed with an acoustic decoder to generate synthetic acoustic data having a voice quality different from that of the acoustic data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-06554 Summary of the Invention [Problem to be solved by the invention]

[0004] It is assumed that the voice quality of an audio signal is converted by a plug-in added to music production software (DAW: Digital Audio Workstation). When the voice quality of multiple audio signals set on different tracks of the music production software is converted and played in parallel, if musical parameters such as pronunciation timing or pitch are completely consistent between the converted multiple audio signals, the reproduced audio will give an audibly unnatural impression. In consideration of the above circumstances, one aspect of the present disclosure aims to generate a reproduced audio that is audibly natural in the voice quality conversion of audio signals set on multiple tracks. [Means for solving the problem]

[0005] In order to solve the above problems, an audio processing method according to one embodiment of the present disclosure assigns to each of a plurality of tracks in music production software an audio processing plug-in including voice quality conversion of an audio signal set for that track, and differentiates a first musical parameter related to a first audio signal after processing by the audio processing plug-in assigned to a first track among the plurality of tracks from a second musical parameter related to a second audio signal after processing by the audio processing plug-in assigned to a second track different from the first track.

[0006] An audio processing system according to one embodiment of the present disclosure includes an assignment unit that assigns an audio processing plug-in including voice quality conversion of an audio signal set on each of a plurality of tracks in music production software, and an adjustment unit that differentiates a first musical parameter related to a first audio signal after processing by the audio processing plug-in assigned to a first track among the plurality of tracks from a second musical parameter related to a second audio signal after processing by the audio processing plug-in assigned to a second track different from the first track.

[0007] A program according to one embodiment of the present disclosure causes a computer system to function as an assignment unit that assigns, to each of a plurality of tracks in music production software, an audio processing plug-in including voice quality conversion of an audio signal set for that track, and an adjustment unit that differentiates a first musical parameter related to a first audio signal after processing by the audio processing plug-in assigned to a first track among the plurality of tracks from a second musical parameter related to a second audio signal after processing by the audio processing plug-in assigned to a second track different from the first track.

[0008] One embodiment of the audio processing software of the present disclosure includes an audio processing plug-in assigned to each of a plurality of tracks in music production software, the audio processing plug-in including voice quality conversion of an audio signal set for the track, and an adjustment program that differentiates a first musical parameter related to a first audio signal after processing by the audio processing plug-in assigned to a first track of the plurality of tracks from a second musical parameter related to a second audio signal after processing by the audio processing plug-in assigned to a second track different from the first track. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating a configuration of a voice processing system according to a first embodiment. [Diagram 2] FIG. 2 is an explanatory diagram of information stored in a storage device. [Diagram 3] FIG. 2 is a schematic diagram of a music production screen. [Figure 4] FIG. 2 is an explanatory diagram of a voice process executed by the voice processing system. [Diagram 5] FIG. 2 is a block diagram illustrating a functional configuration of the voice processing system. [Figure 6] 2 is a block diagram illustrating a configuration of an audio processing unit. FIG. [Figure 7] 11 is an explanatory diagram of a process in which an adjustment unit adjusts sound production timing. FIG. [Figure 8] 13 is a flowchart of a process for generating an audio signal. [Figure 9] 13A to 13C are explanatory diagrams of a process in which an adjustment unit in the second embodiment adjusts the pitch. [Figure 10] 13 is an explanatory diagram of a process in which an adjustment unit of the third embodiment adjusts the volume. FIG. [Figure 11] 13 is an explanatory diagram of a process in which an adjustment unit of a fourth embodiment adjusts characteristic variation parameters. FIG. [Figure 12] FIG. 13 is an explanatory diagram of adjustment conditions in the fifth embodiment. [Figure 13] FIG. 13 is a schematic diagram of a display screen in the sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] A: First embodiment 1 is a block diagram illustrating a configuration of a voice processing system 100 in the first embodiment. The voice processing system 100 is a computer system that executes various types of voice processing, and is realized by an information device such as a smartphone, a tablet terminal, or a personal computer. The voice processing system 100 includes a control device 11, a storage device 12, a display device 13, an operation device 14, and a sound emitting device 15. The voice processing system 100 may be realized as a single device, or may be realized as a plurality of devices configured separately from each other.

[0011] The control device 11 is composed of one or more processors that control each element of the audio processing system 100. For example, the control device 11 is composed of one or more types of processors such as a central processing unit (CPU), a graphics processing unit (GPU), a sound processing unit (SPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC).

[0012] The storage device 12 is one or more memories that store the programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is configured with a known recording medium such as a magnetic recording medium or a semiconductor recording medium. The storage device 12 may be configured with a combination of multiple types of recording media. In addition, a portable recording medium that is detachable from the voice processing system 100, or a recording medium (e.g., cloud storage) to which the control device 11 can write or read via a communication network may be used as the storage device 12.

[0013] FIG. 2 is an explanatory diagram of information stored in the storage device 12. The storage device 12 stores an audio signal X, music production software Qa, an audio processing plug-in Qb, and an adjustment program Qc. The audio signal X is a time domain signal representing an audio waveform. Specifically, the audio signal X is a signal representing the singing voice of a song. For example, an audio signal X generated by sound collection by a sound collection device is stored in the storage device 12. An audio signal X received from an external device via a communication network, or an audio signal X read from a portable recording medium such as an optical disk, may be stored in the storage device 12. An audio signal X generated by known voice synthesis may be stored in the storage device 12.

[0014] The music production software Qa is a program (DAW: Digital Audio Workstation) used for music production (DTM: Desk Top Music). The audio processing plug-in Qb is an auxiliary program for adding extended functions to the music production software Qa. The audio processing plug-in Qb is, for example, a VST (Virtual Studio Technology) plug-in ("VST" is a registered trademark). The audio processing plug-in Qb of the first embodiment realizes audio processing including voice quality conversion of the audio signal X. Voice quality conversion is signal processing that changes the voice quality (tone) while maintaining the spoken content of the voice represented by the audio signal X. The adjustment program Qc is software for adjusting music parameters. For example, audio processing software including the audio processing plug-in Qb and the adjustment program Qc is provided (for example, sold) to a user.

[0015] The display device 13 in Fig. 1 displays an image under the control of the control device 11. The display device 13 is configured with a display panel such as a liquid crystal panel or an organic EL (Electroluminescence) panel. The operation device 14 is an input device that receives instructions from a user. The operation device 14 is, for example, an operator operated by the user, or a touch panel that detects contact by the user. Note that the display device 13 or the operation device 14, which are separate from the voice processing system 100, may be connected to the voice processing system 100 by wire or wirelessly.

[0016] 3 is a schematic diagram of a music production screen 20 displayed on the display device 13 by the music production software Qa. The music production screen 20 displays multiple tracks T (T0, T1, T2) used in music production. Each track T is a processing unit that is played in parallel with each other. The user can set various musical materials such as an audio signal X or music data to each track T by operating the operation device 14. FIG. 3 illustrates a state in which a common audio signal X is set to each of tracks T0, T1, and T2 among the multiple tracks T.

[0017] A user can assign an audio processing plug-in Qb to a desired track T by operating the operation device 14. In the following explanation, an example is shown in which an audio processing plug-in Qb is assigned to each of tracks T1 and T2. Audio processing is performed on the audio signal X of each track Tn (n=1, 2) to which an audio processing plug-in Qb is assigned by the audio processing plug-in Qb. Note that track T1 is an example of a "first track" and track T2 is an example of a "second track."

[0018] 4 is an explanatory diagram of the audio processing executed by the audio processing system 100. By executing voice conversion by the audio processing plug-in Qb on the audio signal X of the track T1, an audio signal Y1 having a voice quality different from that of the audio signal X is generated. Similarly, by executing voice conversion by the audio processing plug-in Qb on the audio signal X of the track T2, an audio signal Y2 having a voice quality different from that of the audio signal X is generated. The voice quality of the audio signal Y1 is different from the voice quality of the audio signal Y2. The audio signal Y1 is an example of a "first audio signal", and the audio signal Y2 is an example of a "second audio signal".

[0019] The audio signal X on track T0, the audio signal Y1 on track T1 after voice conversion, and the audio signal Y2 on track T2 after voice conversion are added together to generate the audio signal Z. Therefore, the audio signal Z represents a voice that is a mixture of three voices with different voice qualities. The voice represented by the audio signal Z can also be expressed as a unison chorus sound in which multiple singers with different voice qualities sing a common melody.

[0020] 3, a play button 21 and a stop button 22 are displayed on the music production screen 20. The play button 21 is an operator that allows the user to instruct the start of playback of the audio signal Z. The stop button 22 is an operator that allows the user to instruct the stop of playback of the audio signal Z.

[0021] The sound emitting device 15 in FIG. 1 reproduces sound under the control of the control device 11. The sound emitting device 15 is an output device such as a speaker or a headphone. Specifically, the sound emitting device 15 reproduces sound represented by an audio signal Z. That is, the audio signal X, the audio signal Y1, and the audio signal Y2 are reproduced in parallel with each other. The sound emitting device 15 may include a D / A converter that converts the audio signal Z from digital to analog, and an amplifier that amplifies the audio signal Z. The sound emitting device 15, which is separate from the audio processing system 100, may be connected to the audio processing system 100 by wire or wirelessly.

[0022] 3, an operation screen Gn is displayed on the display device 13 for each track Tn (T1, T2) to which an audio processing plug-in Qb is assigned. The operation screen Gn is displayed by the function of the audio processing plug-in Qb. Each operation screen Gn is an image including a voice quality list L and a plurality of operators 23 (23a, 23b, 23c, 23d, 23e).

[0023] The voice quality list L is a list of multiple candidates for the voice quality after conversion by voice quality conversion. Considering that each singer has a unique voice quality, the voice quality list L can also be expressed as a list of multiple singers who sing with different voice qualities. The user operates the operation device 14 to select a post-conversion voice quality from the multiple candidates in the voice quality list L. In other words, voice quality conversion using the voice processing plug-in Qb is a process of converting the voice quality of the voice signal X into a voice quality selected by the user from the multiple voice qualities in the voice quality list L. With the above configuration, the voice processing plug-in Qb can generate a voice signal Yn with various voice qualities that reflect the user's intentions or preferences.

[0024] In the following description, it is assumed that a first voice quality is selected for track T1 and a second voice quality is selected for track T2. The first voice quality and the second voice quality are different voice qualities. Voice conversion using the voice processing plug-in Qb assigned to track T1 is a process of converting the voice quality of the audio signal X to the first voice quality. Voice conversion using the voice processing plug-in Qb assigned to track T2 is a process of converting the voice quality of the audio signal X to the second voice quality.

[0025] Each of the multiple operators 23 is a software button for adjusting the acoustic characteristics for each track T. The operator 23a is an operator for adjusting the pitch of the audio signal Yn, and the operator 23b is an operator for adjusting the volume (power, dynamics) of the audio signal Yn. The operator 23c is an operator for adjusting the amplification factor of the audio signal X before voice conversion, and the operator 23d is an operator for adjusting the amplification factor of the audio signal Yn after voice conversion. The operator 23e is an operator for adjusting the ratio (mixing ratio) of the audio signal Yn in the finally generated audio signal Z. The user can adjust the voice quality and acoustic characteristics after voice conversion for each track T to which the audio processing plug-in Qb is assigned, by operating each operation screen Gn using the operation device 14.

[0026] 5 is a block diagram illustrating a functional configuration of the audio processing system 100. The control device 11 executes programs (music production software Qa, audio processing plug-in Qb, and adjustment program Qc) stored in the storage device 12 to realize a plurality of functions (allocation unit 31, audio processing unit 32-1, audio processing unit 32-2, signal synthesis unit 33) for processing the audio signal X. The allocation unit 31 and the signal synthesis unit 33 are realized by, for example, the music production software Qa.

[0027] The allocation unit 31 allocates an audio processing plug-in Qb to each track Tn in the music production software Qa. Specifically, the allocation unit 31 allocates the audio processing plug-in Qb designated by the user to the track Tn designated by the user among the multiple tracks T on the music production screen 20.

[0028] The audio processing unit 32-n is realized by the audio processing plug-in Qb for each track Tn to which the audio processing plug-in Qb is assigned. The audio processing unit 32-n generates an audio signal Yn by audio processing on the audio signal X. Specifically, the audio processing unit 32-1 generates an audio signal Y1 of a first voice quality from the audio signal X, and the audio processing unit 32-2 generates an audio signal Y2 of a second voice quality from the audio signal X.

[0029] The signal synthesis unit 33 generates an audio signal Z by adding an audio signal X of a track T0, an audio signal Y1 of a track T1 generated by an audio processing unit 32-1, and an audio signal Y2 of a track T2 generated by an audio processing unit 32-2. The mixing ratio of the audio signal Yn to the audio signal X is set to a value designated by the operator 23e on the operation screen Gn.

[0030] 6 is a block diagram illustrating the configuration of each audio processing unit 32-n. The audio processing unit 32-n includes a signal acquisition unit 41, a signal analysis unit 42, an adjustment unit 43, a voice quality conversion unit 44, and a signal output unit 45. The signal acquisition unit 41 acquires an audio signal X from the storage device 12. The signal acquisition unit 41 amplifies the audio signal X by an amplification factor specified by the operator 23c on the operation screen Gn. The signal acquisition unit 41 may be omitted.

[0031] The signal analysis unit 42 processes the audio signal X to generate a time series of control data A. Each piece of control data A is generated for each unit period obtained by dividing the audio signal X on the time axis. That is, the signal analysis unit 42 generates control data A for each unit period from a sample series within the audio signal X. The control data A includes phoneme symbols and a plurality of musical parameters P (Pa, Pb, Pc, ...). The phoneme symbols are codes for identifying the phonemes that make up the audio signal X. That is, the phoneme symbols represent the pronunciation of the voice represented by the audio signal X (i.e., the lyrics).

[0032] The music parameters P are parameters that represent the acoustic characteristics of the audio signal X. The control data A of the first embodiment includes a pronunciation timing Pa, a pitch Pb, and a volume Pc as the music parameters P. The pronunciation timing Pa is the timing at which a series of pronunciations in a singing voice starts. The pitch Pb is the fundamental frequency (pitch) of the audio signal X. The volume Pc is the strength of the pronunciation in the audio signal X.

[0033] A trained generative model Ma is used for generating the control data A by the signal analysis unit 42. The generative model Ma is a statistical model that has learned the relationship between the audio signal X and the control data A by prior machine learning, and is included in the audio processing plug-in Qb. The signal analysis unit 42 generates the control data A by processing the audio signal X of each unit period using the generative model Ma.

[0034] The generative model Ma is constructed by supervised machine learning using multiple training data. Each training data includes a training speech signal X and training control data A. The control data A of each training data is ground truth data that represents the acoustic features of the speech signal X of the training data.

[0035] A plurality of variables defining the generative model Ma are iteratively updated so that the control data A generated by the provisional generative model Ma from the training speech signal X and the learning control data A corresponding to the speech signal X are mutually approximated. Therefore, the trained generative model Ma outputs control data A that is statistically valid for the unknown speech signal X under the underlying relationship between the speech signal X and the control data A in the plurality of training data.

[0036] The adjustment unit 43 generates control data Bn by adjusting the control data A. For example, the adjustment unit 43 adjusts the pitch Pb of the control data A in response to an instruction to the operator 23a of the operation screen Gn, and adjusts the volume Pc of the control data A in response to an instruction to the operator 23b. The adjustment unit 43 also adjusts the sound timing Pa of the control data A. The adjustment of the sound timing Pa will be described later. The adjustment unit 43 is realized by the adjustment program Qc. The function (adjustment unit 43) realized by the adjustment program Qc may be included in the audio processing plug-in Qb. The control data A is an example of "first control data", and the control data Bn is an example of "second control data".

[0037] The control data Bn includes each musical parameter P (Pa, Pb, Pc, ...) after adjustment by the adjustment unit 43. That is, the control data Bn includes a plurality of musical parameters P that represent the acoustic characteristics of the audio signal Yn. As can be understood from the above example, the control data A includes the initial values ​​of each musical parameter P, and the control data Bn includes each musical parameter P that has been changed from its initial value. The musical parameter P of the control data B1 is an example of a "first musical parameter," and the musical parameter P of the control data B2 is an example of a "second musical parameter."

[0038] Since the control data A is generated from a common audio signal X, the control data A is substantially common between tracks T1 and T2. On the other hand, the adjustment by the adjustment unit 43 is performed individually for each track Tn in response to an operation on the operation screen Gn, so the control data Bn differs between tracks T1 and T2. That is, the adjustment unit 43 makes the music parameter P (control data B1) of the audio signal Y1 and the music parameter P (control data B2) of the audio signal Y2 different.

[0039] The voice conversion unit 44 processes the control data Bn after processing by the adjustment unit 43 to generate a voice signal Yn. Specifically, the voice conversion unit 44 generates a sample sequence within each unit period of the voice signal Yn from the control data Bn for that unit period. The voice signal Yn is generated by connecting the sample sequences of adjacent unit periods to each other. The generation of the voice signal Yn by the voice conversion unit 44 can also be expressed as voice synthesis that synthesizes a voice instructed by the control data Bn.

[0040] A trained generative model Mb is used to generate the voice signal Yn by the voice conversion unit 44. The generative model Mb is a statistical model that has learned the relationship between the control data Bn and the voice signal Yn in advance by machine learning. The voice conversion unit 44 processes the control data Bn of each unit period using the generative model Mb to generate the voice signal Yn.

[0041] The voice processing plug-in Qb of the first embodiment includes a generation model Mb corresponding to each of a plurality of voice qualities included in the voice quality list L. That is, a separate generation model Mb is prepared for each voice quality. The voice quality conversion unit 44 generates a voice signal Yn by using a generation model Mb corresponding to a voice quality selected by a user for a track Tn, among a plurality of generation models Mb corresponding to different voice qualities. Note that after the voice processing plug-in Qb is introduced into the voice processing system 100, a generation model Mb corresponding to an additional voice quality may be added to the voice processing plug-in Qb.

[0042] The generative model Mb is constructed by supervised machine learning using multiple training data. Multiple sets of training data are prepared for each of multiple voice qualities. Each training data includes training control data B and training speech signal Y. The speech signal Y of each training data is ground truth data representing the waveform of speech to be generated from the control data B of the training data.

[0043] A number of variables defining the generative model Mb are iteratively updated so that the speech signal Y generated by the provisional generative model Mb from the training control data B and the speech signal Y corresponding to the control data B are mutually approximated. Thus, the trained generative model Mb outputs a statistically valid speech signal Yn for unknown control data Bn, based on the underlying relationship between the control data B and the speech signal Y in the multiple training data. The generative model Mb corresponding to each sound quality generates a speech signal Yn representing the sound of that sound quality.

[0044] The signal output unit 45 outputs the audio signal Yn to the signal synthesis unit 33. The signal output unit 45 also amplifies the audio signal Yn by an amplification factor specified by the operator 23d on the operation screen Gn. As described above, a plurality of audio signals Yn corresponding to different voice qualities are generated from a common audio signal X. The signal output unit 45 may be omitted.

[0045] 7 is an explanatory diagram of the process in which the adjustment unit 43 adjusts the sounding timing Pa. As described above, in each sound processing unit 32-n, the control data A is generated from a common sound signal X. Therefore, the sounding timing Pa of the control data A generated by the signal analysis unit 42 of the sound processing unit 32-1 and the sounding timing Pa of the control data A generated by the signal analysis unit 42 of the sound processing unit 32-2 match the sounding timing of the sound signal X. As illustrated in FIG. 7, the adjustment unit 43 of each sound processing unit 32-n changes the sounding timing Pa specified by the control data A and generates control data Bn that specifies the changed sounding timing Pa.

[0046] Specifically, the adjustment unit 43 generates a random number within a predetermined range, and moves the sounding timing Pa specified by the control data A forward or backward according to the random number. The control data Bn specifies the sounding timing Pa after the movement according to the random number. The change of the sounding timing Pa is performed individually for each sound processing unit 32-n. Therefore, as can be understood from FIG. 7, the sounding timing Pa of the sound signal Y1 differs from the sounding timing Pa of the sound signal Y2. Furthermore, the sounding timing Pa of each sound signal Yn also differs from the sounding timing of the sound signal X.

[0047] That is, the adjustment unit 43 (adjustment program Qc) of the first embodiment makes the pronunciation timing Pa for the audio signal Y1 after voice quality conversion by the audio processing plug-in Qb assigned to track T1 different from the pronunciation timing Pa for the audio signal Y2 after voice quality conversion by the audio processing plug-in Qb assigned to track T2.

[0048] As described above, in the first embodiment, the sound generation timing Pa differs between the audio signal Y1 processed by the audio processing plug-in Qb assigned to the track T1 and the audio signal Y2 processed by the audio processing plug-in Qb assigned to the track T2. Therefore, compared to a form in which the sound generation timing Pa coincides between the audio signal Y1 and the audio signal Y2, a sound that sounds more natural to the ears can be generated.

[0049] 8 is a flowchart showing a process in which the control device 11 generates an audio signal Z from an audio signal X. The process in FIG. 8 starts when the user operates the play button 21.

[0050] When the process starts, the control device 11 determines (S1) whether or not the audio processing plug-in Qb is assigned to multiple tracks T. If the audio processing plug-in Qb is not assigned to multiple tracks T (S1: NO), the control device 11 executes normal processing provided by the music production software Qa without executing the following processes (S2 to S10).

[0051] When the audio processing plug-in Qb is assigned to multiple tracks T (S1: YES), the control device 11 accepts an operation from the user on each operation screen Gn (S2). Specifically, the control device 11 accepts an operation to select a desired voice quality from the voice quality list L of each operation screen Gn, or an operation on the operator 23 of each operation screen Gn from the user.

[0052] The control device 11 generates an audio signal Yn from the audio signal X for each track Tn to which the audio processing plug-in Qb is assigned (S3 to S7). That is, a plurality of audio signals Yn (Y1, Y2) corresponding to different voice qualities are generated from the audio signal X.

[0053] First, the control device 11 (signal acquisition unit 41) acquires a portion of the audio signal X stored in the storage device 12 that corresponds to the current unit period (S3). The control device 11 amplifies the audio signal X by an amplification factor instructed by an operation on the operator 23c on the operation screen Gn.

[0054] The control device 11 (signal analysis unit 42) processes the voice signal X using the trained generative model Ma to generate control data A (S4). The control device 11 (adjustment unit 43) adjusts the control data A to generate control data Bn (S5). The adjustment of the control data A includes a process of making the pronunciation timing Pa different between the control data B1 and the control data B2. The control device 11 (voice quality conversion unit 44) processes the control data Bn using the trained generative model Mb to generate a voice signal Yn (S6). The control device 11 (signal output unit 45) amplifies the voice signal Yn by an amplification factor instructed by an operation on the operator 23d of the operation screen Gn (S7).

[0055] As described above, voice quality conversion using the voice processing plug-in Qb includes a process (S4) of generating control data A by processing the voice signal X using a trained generative model Ma, a process (S5) of generating control data Bn by performing a process on the control data A that differs the musical parameter P, and a process (S6) of generating a voice signal Yn by processing the control data Bn using a trained generative model Mb.

[0056] The control device 11 (signal synthesis unit 33) generates an audio signal Z by adding the audio signal X of the track T0, the audio signal Y1 of the track T1, and the audio signal Y2 of the track T2 (S8). The control device 11 (signal synthesis unit 33) outputs the audio signal Z to the sound emitting device 15 (S9). Therefore, a mixed sound (i.e., a chorus sound) of three types of voices singing a common song with different vocal qualities is reproduced by the sound emitting device 15.

[0057] The control device 11 judges whether or not a predetermined end condition is met (S10). The end condition is, for example, when an operation on the stop button 22 is accepted from the user, or when processing is completed for the entire section of the audio signal X. If the end condition is not met (S10: NO), the control device 11 shifts the processing to step S2. That is, the acceptance of an operation on the operation screen Gn (S2), the voice quality conversion by the audio processing plug-in Qb (S3 to S7), and the playback of the audio signal Z (S8, S9) are repeated until the end condition is met.

[0058] That is, in the first embodiment, the selection of one of a plurality of voice qualities (S2) and the voice quality conversion by the voice processing plug-in Qb (S3 to S7) are executed in real time in parallel with the playback of the voice signal Z after the voice quality conversion (S8, S9). Therefore, at any point in the process of playing the voice signal Z, the voice quality of the voice signal Yn is changed as needed in response to an instruction from the user on the operation screen Gn. When the end condition is met (S10: YES), the control device 11 ends the processing of FIG. 8.

[0059] In the first embodiment, a process (S4) of generating a time series of control data A from the voice signal X using a generation model Ma, and a process (S6) of generating a voice signal Yn after voice conversion from control data Bn using a generation model Mb are executed sequentially. Therefore, compared to a configuration in which voice conversion is realized by voice processing that directly processes the voice signal X, it is possible to generate a voice that is auditorily natural. Also, as a result of processing (S5) on the intermediate control data A, the music parameters (pronunciation timing Pa) are different between the voice signals Y1 and Y2. Therefore, the process for making the music parameters different is simplified compared to a form in which the music parameters are made different by voice processing on the voice signals Y1 and Y2 after voice conversion.

[0060] B: Second embodiment A second embodiment will be described. Note that, for elements having the same functions as those in the first embodiment in each of the following exemplary aspects, the same reference numerals as those in the first embodiment will be used, and detailed descriptions of each will be omitted as appropriate.

[0061] In the first embodiment, the adjustment unit 43 of the audio processing unit 32-n adjusts the sounding timing Pa of the audio signal Yn. In the second embodiment, the adjustment unit 43 of the audio processing unit 32-n adjusts the pitch Pb of the audio signal Yn. Operations other than the operation of the adjustment unit 43 are the same as in the first embodiment. Therefore, the second embodiment also achieves the same effects as the first embodiment.

[0062] Fig. 9 is an explanatory diagram of the process in which the adjustment unit 43 of the second embodiment adjusts the pitch Pb. As described above, in each audio processing unit 32-n, the control data A is generated from a common audio signal X. Therefore, the pitch Pb of the control data A generated by the signal analysis unit 42 of the audio processing unit 32-1 and the pitch Pb of the control data A generated by the signal analysis unit 42 of the audio processing unit 32-2 match the pitch of the audio signal X. As illustrated in Fig. 9, the adjustment unit 43 of each audio processing unit 32-n changes the pitch Pb specified by the control data A and generates control data Bn that specifies the changed pitch Pb.

[0063] Specifically, the adjustment unit 43 generates a random number within a predetermined range, and changes the pitch Pb designated by the control data A to a higher or lower pitch according to the random number. The control data Bn designates the pitch Pb after the change according to the random number. The change in pitch Pb is performed individually for each audio processing unit 32-n. Therefore, as can be seen from FIG. 9, the pitch Pb of the audio signal Y1 differs from the pitch Pb of the audio signal Y2. The difference between the pitch Pb of the audio signal Y1 and the pitch Pb of the audio signal Y2 is less than the pitch difference of a semitone in equal temperament, for example. The pitch Pb of each audio signal Yn also differs from the pitch of the audio signal X.

[0064] As illustrated above, the adjustment unit 43 of the second embodiment differentiates the pitch Pb of the audio signal Y1 after voice quality conversion using the audio processing plug-in Qb assigned to track T1 from the pitch Pb of the audio signal Y2 after voice quality conversion using the audio processing plug-in Qb assigned to track T2.

[0065] However, in a configuration in which the pitch Pb of the audio signal Y1 and the audio signal Y2 are completely the same, the reproduced audio of the audio signal Z may give an audibly unnatural impression (e.g., an artificial or mechanical impression). In the second embodiment, since the pitch Pb is made different between the audio signal Y1 and the audio signal Y2, even if a common audio signal X is set for the track T1 and the track T2 as described above, it is possible to generate an audibly natural reproduced audio in which the pitch Pb of the audio signal Y1 and the audio signal Y2 do not match.

[0066] C: Third embodiment The adjustment unit 43 of the audio processing unit 32-n in the third embodiment adjusts the volume Pc of the audio signal Yn. Operations other than the operation of the adjustment unit 43 are similar to those in the first embodiment. Therefore, the third embodiment also achieves the same effects as the first embodiment.

[0067] Fig. 10 is an explanatory diagram of a process in which the adjustment unit 43 of the third embodiment adjusts the volume Pc. As described above, in each audio processing unit 32-n, the control data A is generated from a common audio signal X. Therefore, the volume Pc of the control data A generated by the signal analysis unit 42 of the audio processing unit 32-1 and the volume Pc of the control data A generated by the signal analysis unit 42 of the audio processing unit 32-2 match the volume of the audio signal X. As illustrated in Fig. 10, the adjustment unit 43 of each audio processing unit 32-n changes the volume Pc specified by the control data A and generates control data Bn that specifies the changed volume Pc.

[0068] Specifically, the adjustment unit 43 generates a random number within a predetermined range, and changes the volume Pc designated by the control data A to a higher or lower volume according to the random number. The control data Bn designates the volume Pc after the change according to the random number. The change in volume Pc is performed individually for each audio processing unit 32-n. Therefore, as can be seen from FIG. 10, the volume Pc of the audio signal Y1 is different from the volume Pc of the audio signal Y2. Furthermore, the volume Pc of each audio signal Yn is also different from the volume of the audio signal X.

[0069] As illustrated above, the adjustment unit 43 of the third embodiment makes the volume Pc for the audio signal Y1 after voice quality conversion using the audio processing plug-in Qb assigned to track T1 different from the volume Pc for the audio signal Y2 after voice quality conversion using the audio processing plug-in Qb assigned to track T2.

[0070] However, in a configuration in which the volume Pc of the audio signal Y1 and the audio signal Y2 are completely the same, the reproduced audio of the audio signal Z may give an audibly unnatural impression (e.g., an artificial or mechanical impression). In the third embodiment, since the volume Pc is made different between the audio signal Y1 and the audio signal Y2, even if a common audio signal X is set for the track T1 and the track T2 as described above, it is possible to generate an audibly natural reproduced audio in which the volume Pc does not match between the audio signal Y1 and the audio signal Y2.

[0071] D: Fourth embodiment The control data A of the fourth embodiment includes a parameter related to the temporal variation of an acoustic characteristic (hereinafter referred to as a "characteristic variation parameter Pd") as a music parameter P. The temporal variation of an acoustic characteristic is a vibrato in which the pitch of a voice varies repeatedly. Therefore, the characteristic variation parameter Pd is a parameter related to the vibrato of the voice. Specifically, the characteristic variation parameter Pd is, for example, the period, amplitude or phase of the vibrato. Note that the temporal variation of an acoustic characteristic may also be a tremolo in which the volume of a voice varies repeatedly.

[0072] The adjustment unit 43 of the audio processing unit 32-n in the fourth embodiment adjusts the characteristic variation parameter Pd of the audio signal Yn. Operations other than the operation of the adjustment unit 43 are similar to those in the first embodiment. Therefore, the fourth embodiment also achieves the same effects as the first embodiment.

[0073] Fig. 11 is an explanatory diagram of a process in which the adjustment unit 43 of the fourth embodiment adjusts the characteristic variation parameter Pd. As described above, in each audio processing unit 32-n, the control data A is generated from a common audio signal X. Therefore, the characteristic variation parameter Pd of the control data A generated by the signal analysis unit 42 of the audio processing unit 32-1 and the characteristic variation parameter Pd of the control data A generated by the signal analysis unit 42 of the audio processing unit 32-2 match the characteristic variation parameter of the audio signal X. As illustrated in Fig. 11, the adjustment unit 43 of each audio processing unit 32-n changes the characteristic variation parameter Pd specified by the control data A and generates control data Bn that specifies the changed characteristic variation parameter Pd.

[0074] Specifically, the adjustment unit 43 generates a random number within a predetermined range, and changes the characteristic variation parameter Pd specified by the control data A in accordance with the random number. The control data Bn specifies the characteristic variation parameter Pd after the change in accordance with the random number. The change of the characteristic variation parameter Pd is performed individually for each audio processing unit 32-n. Therefore, as can be seen from FIG. 11, the characteristic variation parameter Pd of the audio signal Y1 differs from the characteristic variation parameter Pd of the audio signal Y2. Furthermore, the characteristic variation parameter Pd of each audio signal Yn also differs from the characteristic variation parameter Pd of the audio signal X.

[0075] As illustrated above, the adjustment unit 43 of the fourth embodiment differentiates the characteristic variation parameter Pd for the audio signal Y1 after voice quality conversion using the audio processing plug-in Qb assigned to track T1 from the characteristic variation parameter Pd for the audio signal Y2 after voice quality conversion using the audio processing plug-in Qb assigned to track T2.

[0076] However, in a configuration in which the characteristic variation parameter Pd is completely identical between the audio signals Y1 and Y2, the reproduced audio of the audio signal Z may give an audibly unnatural impression (e.g., an artificial or mechanical impression). In the fourth embodiment, since the characteristic variation parameter Pd is made different between the audio signals Y1 and Y2, even if a common audio signal X is set for the tracks T1 and T2 as described above, it is possible to generate an audibly natural reproduced audio in which the characteristic variation parameter Pd does not match between the audio signals Y1 and Y2.

[0077] In the first embodiment, the sounding timing Pa is adjusted, in the second embodiment, the pitch Pb is adjusted, in the third embodiment, the volume Pc is adjusted, and in the fourth embodiment, the characteristic variation parameter Pd is adjusted. Two or more forms selected from the above examples may be appropriately combined. That is, the adjustment unit 43 of the audio processing unit 32-n is comprehensively expressed as an element that makes the music parameter P of the audio signal Y1 different from the music parameter P of the audio signal Y2. The adjustment program Qc is a program that makes the music parameter P of the audio signal Y1 different from the music parameter P of the audio signal Y2. And the music parameter P includes at least one of the sounding timing Pa, the pitch Pb, the volume Pc, and the characteristic variation parameter Pd, for example.

[0078] E: Fifth embodiment The adjustment unit 43 of the fifth embodiment adjusts the music parameter P of the control data A under a predetermined condition (hereinafter referred to as the "adjustment condition"). The adjustment condition is a condition for adjusting the music parameter P of each audio signal Yn to make the music parameter P different between the audio signal Y1 and the audio signal Y2. For example, the direction (increase / decrease) or the amount of change (range width of the fluctuation range) of the music parameter P is exemplified as the adjustment condition.

[0079] The audio processing plug-in Qb individually specifies adjustment conditions for each of a plurality of voice qualities. For example, the adjustment conditions corresponding to the first voice quality are different from the adjustment conditions corresponding to the second voice quality. Therefore, when different voice qualities are selected between track T1 and track T2, the adjustment conditions of the music parameters P of the audio signal Y1 corresponding to track T1 are different from the adjustment conditions of the music parameters P of the audio signal Y2 corresponding to track T2. That is, the adjustment unit 43 adjusts the music parameters P of the control data A under the adjustment conditions that differ for each voice quality.

[0080] FIG. 12 is an explanatory diagram of adjustment conditions related to the pronunciation timing Pa. In FIG. 12, a variation range V (V1, V2) related to the pronunciation timing Pa is illustrated as an adjustment condition. Each variation range V is a range set based on the pronunciation timing of the audio signal X before voice conversion. The variation range V1 is an adjustment condition corresponding to the first voice quality selected for the track T1. That is, the pronunciation timing Pa of the audio signal Y1 after voice conversion is located within the variation range V1. On the other hand, the variation range V2 is an adjustment condition corresponding to the second voice quality selected for the track T2. That is, the pronunciation timing Pa of the audio signal Y2 after voice conversion is located within the variation range V2.

[0081] 12, the variation range V1 corresponding to the first voice quality is different from the variation range V2 corresponding to the second voice quality. For example, the positions on the time axis and the range widths of the variation ranges V1 and V2 are different. Specifically, the variation range V1 is located earlier (in the past) than the variation range V2, and the variation range V1 is wider than the variation range V2.

[0082] Therefore, the pronunciation timing Pa of the audio signal Y1 of the first voice quality is likely to be located before the pronunciation timing Pa of the audio signal Y2 of the second voice quality. That is, the audio signal Y1 tends to start pronouncing earlier than the pronunciation timing of the audio signal X, and the audio signal Y2 tends to start pronouncing later than the pronunciation timing of the audio signal X.

[0083] The fifth embodiment also achieves the same effects as the above-mentioned embodiments. In the fifth embodiment, the adjustment conditions of the music parameter P are different between the audio signal Y1 of the first voice quality and the audio signal Y2 of the second voice quality. Therefore, it is possible to generate the audio signal Y1 reflecting the pronunciation tendency specific to the first voice quality and the audio signal Y2 reflecting the pronunciation tendency specific to the second voice quality.

[0084] Although FIG. 12 focuses on the sounding timing Pa for the sake of convenience, adjustment conditions may also be set individually for each voice quality for other musical parameters P such as pitch Pb, volume Pc, or characteristic variation parameter Pd.

[0085] F: Sixth embodiment 13 is a schematic diagram of a screen displayed on the display device 13 in the sixth embodiment. In the sixth embodiment, in addition to the music production screen 20, a voice quality list L1, a voice quality list L2, and an operation screen G are displayed on the display device 13. The voice quality list L2 corresponds to the voice quality list L on the operation screen Gn in the first embodiment. In other words, the voice quality list L1 is an image for selecting a voice quality for track T1, and the voice quality list L2 is an image for selecting a voice quality for track T2.

[0086] The operation screen G is an image for adjusting the acoustic characteristics of the tracks T1 and T2, and includes a plurality of operators 23 (23a, 23b, 23c, 23d, 23e) like each operation screen Gn in the first embodiment. That is, in the sixth embodiment, a common operation screen G is displayed for a plurality of tracks T. In response to a user's operation on each operator 23 of the operation screen G, the acoustic characteristics are set commonly for both the tracks T1 and T2. That is, the user can adjust the acoustic characteristics for both the tracks T1 and T2 in response to an operation on the operation screen G. This reduces the burden on the user who edits a plurality of tracks Tn.

[0087] G: Variation Specific modified embodiments added to each of the above-mentioned embodiments are exemplified below. Two or more embodiments selected from the following examples may be appropriately combined as long as they are not mutually contradictory.

[0088] (1) In the above-mentioned embodiments, the voice conversion includes the generation of the control data A and the generation of the voice signal Yn, but the specific method of the voice conversion for generating the voice signal Yn from the voice signal X is not limited to the above examples. For example, the voice processing unit 32-n may generate the voice signal Yn with the voice converted by directly processing the voice signal X to change the frequency characteristics. In other words, the use of the generation models Ma and Mb may be omitted.

[0089] (2) In each of the above-mentioned embodiments, the voice signal Y1 and the voice signal Y2 have a common melody (unison), but the adjustment unit 43 may adjust the pitch Pb of each voice signal Yn so that the voice signal Y1 and the voice signal Y2 are parallel with a predetermined pitch difference (tone). For example, the adjustment unit 43 adjusts the pitch Pb of each voice signal Yn so that the pitch difference between the voice signal Y1 and the voice signal Y2 is in a musically harmonious relationship (for example, a tone relationship of a third or the like). According to the above-mentioned embodiment, a voice signal Z of a chorus sound in which a plurality of singers with different voice qualities sing different melodies is generated.

[0090] (3) In the above-described embodiments, the audio signal Z is generated by adding the audio signal X, the audio signal Y1, and the audio signal Y2, but the addition of the audio signal X may be omitted. In other words, the signal synthesis unit 33 may generate the audio signal Z by adding the audio signal Y1 and the audio signal Y2.

[0091] (4) In each of the above embodiments, for convenience, attention has been given to two tracks T (T1, T2), but the number of tracks T to which the audio processing plug-in Qb is assigned is arbitrary. For example, the audio processing plug-in Qb may be assigned to each of three or more tracks T.

[0092] (5) In each of the above-described embodiments, the voice quality selection (S2) and voice quality conversion (S3 to S7) are performed in real time in parallel with the playback of the audio signal Z (S8, S9). However, the voice quality selection (S2) and voice quality conversion (S3 to S7) may be performed for the entire section of the audio signal X before the playback of the audio signal Z (S8, S9) is started.

[0093] (6) It is also possible to change the adjustment conditions of the musical parameters P over time within a song. For example, the adjustment unit 43 may make the variation range V of the musical parameters P different between a specific section of the song and other sections. For example, the variation range V of the musical parameters P is reduced to a sufficiently small range at the beginning of a specific phrase in the song, and the variation range V is expanded over time toward the end of the phrase. In addition, for example, the adjustment conditions of the musical parameters P may be set individually for each structural section obtained by dividing the song on the time axis according to musical meaning. The structural sections are, for example, the intro, A verse, B verse, chorus, outro, and the like.

[0094] (7) In the above-described embodiments, the voice processing plug-in Qb for voice conversion is exemplified, but the voice processing realized by the voice processing plug-in Qb is not limited to voice conversion. For example, voice synthesis may be realized by the voice processing plug-in Qb. The voice processing plug-in Qb for voice synthesis processes synthesis data assigned to the track Tn to generate a voice signal Yn representing a synthetic voice.

[0095] Specifically, in a form in which the voice processing plug-in Qb realizes voice synthesis, synthesis data is supplied to each voice processing unit 32-n instead of the voice signal X. The synthesis data is data that specifies the conditions of the voice to be synthesized. For example, the synthesis data specifies the pitch, pronunciation period, and pronunciation character for each of a plurality of notes that constitute a piece of music. The pitch is any one of a plurality of scale notes that are set discretely (for example, a note number). The pronunciation period is specified, for example, by the start point and duration of the note. The pronunciation character is a symbol (for example, a phoneme symbol) that represents the lyrics of the piece of music. The synthesis data is, for example, a music file that complies with the MIDI (Musical Instrument Digital Interface) standard. Note that the pitch may be omitted from the synthesis data. In other words, a voice such as a conversation sound that does not include musical elements may be synthesized.

[0096] Each voice processing unit 32-n generates a voice signal Yn representing a waveform of a voice specified by the synthesis data. Any known voice synthesis is used to generate the voice signal Yn by the voice processing unit 32-n. For example, the voice synthesis by the voice processing unit 32-n may be a voice synthesis of a segment concatenation type that connects a plurality of voice segments, or a voice synthesis of a statistical model type that uses a statistical model such as a deep neural network or an HMM (Hidden Markov Model).

[0097] In a form in which the audio processing plug-in Qb realizes audio synthesis, the adjustment unit 43 makes the music parameter P different between the audio signal Y1 synthesized by the audio processing plug-in Qb assigned to the track T1 and the audio signal Y2 synthesized by the audio processing plug-in Qb assigned to the track T2.

[0098] Furthermore, the voice processing realized by the voice processing plug-in Qb is not limited to voice conversion and voice synthesis. For example, voice processing that imparts an acoustic effect to the voice signal X (hereinafter referred to as "effect imparting") may be realized by the voice processing plug-in Qb. Examples of effect imparting include reverberation that imparts a reverberation effect to the voice signal X, and equalization that adjusts the frequency characteristics of the sound signal X.

[0099] (8) In each of the above-described embodiments, the music parameter P of the audio signal Y1 is made different from the music parameter P of the audio signal Y2 by adjusting the control data A, but the method for making the music parameter P different between the audio signal Y1 and the audio signal Y2 is not limited to the above examples.

[0100] For example, the signal acquisition unit 41 of each audio processing unit 32-n may adjust the music parameter P of the audio signal X for each track Tn individually, thereby making the music parameter P different between the audio signal Y1 and the audio signal Y2. That is, the signal acquisition unit 41 in each of the above-mentioned embodiments may function as the adjustment unit 43 that adjusts the music parameter P. Also, the signal output unit 45 of each audio processing unit 32-n may adjust the music parameter P of the audio signal Yn after voice quality conversion individually for each track Tn. That is, the signal output unit 45 in each of the above-mentioned embodiments may function as the adjustment unit 43 that adjusts the music parameter P. As illustrated above, the process of making the music parameter P different between the audio signal Y1 and the audio signal Y2 may be executed at any time during the process of generating the audio signal Z from the audio signal X.

[0101] (9) In each of the above embodiments, a common audio signal X is set for multiple tracks T (T0, T1, T2), but a separate audio signal X may be set for each of the multiple tracks T. For example, the audio signal Xn of each track Tn (T1, T2) may differ from the audio signal X of track T0 in terms of acoustic features such as pitch, volume, or timbre. Also, the audio signal X1 of track T1 may differ from the audio signal X2 of track T2 in terms of acoustic features.

[0102] (10) The types of musical parameters P are not limited to the information exemplified in the above-mentioned embodiments (pronunciation timing Pa, pitch Pb, volume Pc, characteristic variation parameter Pd). For example, various musical parameters P, such as breathiness, clarity, or mute timing, may be made different between the audio signal Y1 and the audio signal Y2. Breathiness is an index (breathiness) relating to the ratio of breathy components in the voice. Clarity is an index (brightness) of the dominance of harmonic components over non-harmonic components. Mute timing is the timing at which a series of pronunciations in a singing voice ends.

[0103] (11) In the above embodiments, the audio signal (X, Yn, Z) representing a singing voice is processed, but the above embodiments are also applicable to the processing of an audio signal representing any type of musical sound. The sound represented by the audio signal includes not only singing voice but also musical sounds such as musical instrument sounds. In the present disclosure, the term "audio signal" may be replaced with the expression "audio signal" including singing voice and musical sounds.

[0104] (12) The voice processing system 100 may be realized by a server device that communicates with an information device such as a smartphone, a tablet terminal, or a personal computer. For example, the voice processing system 100 generates a voice signal Z by performing voice processing including voice quality conversion on a voice signal X received from the information device, and transmits the voice signal Z to the information device.

[0105] (13) As described above, the functions of the voice processing system 100 exemplified above are realized by the cooperation of one or more processors constituting the control device 11 and the program stored in the storage device 12. The program according to the present disclosure can be provided in a form stored in a computer-readable recording medium and installed in a computer. The recording medium is, for example, a non-transitory recording medium, and a good example is an optical recording medium (optical disk) such as a CD-ROM, but also includes any known type of recording medium such as a semiconductor recording medium or a magnetic recording medium. Note that the non-transitory recording medium includes any recording medium except for a transient, propagating signal, and does not exclude volatile recording media. In addition, in a configuration in which a distribution device distributes a program via a communication network, the storage medium that stores the program in the distribution device corresponds to the non-transitory recording medium described above.

[0106] H: Notes From the above-described exemplary embodiments, the following configurations can be understood, for example.

[0107] An audio processing method according to one aspect (aspect 1) of the present disclosure assigns, to each of a plurality of tracks in music production software, an audio processing plug-in including a voice quality conversion of an audio signal set for that track, and differentiates a first musical parameter related to a first audio signal after processing by the audio processing plug-in assigned to a first track among the plurality of tracks from a second musical parameter related to a second audio signal after processing by the audio processing plug-in assigned to a second track different from the first track. According to the above aspect, since the musical parameters differ between the first audio signal after processing by the audio processing plug-in assigned to the first track and the second audio signal after processing by the audio processing plug-in assigned to the second track, it is possible to generate a reproduced audio that is auditorily natural compared to an aspect in which the musical parameters completely match between the first audio signal and the second audio signal.

[0108] A "track" is a unit of music that is played in parallel with one another. That is, music production software is used to create music in which multiple tracks are played in parallel. A "track" may also be called something else, such as a "clip."

[0109] In a specific example (Aspect 2) of Aspect 1, the first music parameter and the second music parameter include a sounding timing. According to the above aspect, even if a common audio signal is set for the first track and the second track, it is possible to generate a reproduced audio that is auditorily natural in that the sounding timing does not match between the first audio signal and the second audio signal.

[0110] In a specific example (Aspect 3) of Aspect 1 or Aspect 2, the first music parameter and the second music parameter include at least one of pitch, volume, and a parameter related to temporal variation of an acoustic characteristic.

[0111] According to the embodiment in which the pitch is made different between the first audio signal and the second audio signal, even if a common audio signal is set for the first track and the second track, it is possible to generate a reproduced audio that is auditorily natural in that the pitch does not match between the first audio signal and the second audio signal.

[0112] By varying the volume (loudness or dynamics) between the first and second audio signals, even if a common audio signal is set for the first and second tracks, it is possible to generate reproduced audio that is auditorily natural in that the volume does not match between the first and second audio signals.

[0113] According to an embodiment in which parameters relating to the temporal variation of acoustic characteristics are made different between the first audio signal and the second audio signal, even if a common audio signal is set for the first track and the second track, it is possible to generate a reproduced audio that is auditorily natural in that the temporal variations of the acoustic characteristics do not match between the first audio signal and the second audio signal.

[0114] The "acoustic characteristic" is, for example, pitch or volume. Therefore, the "variation of the acoustic characteristic over time" includes, for example, vibrato, in which the pitch of the voice changes repeatedly, or tremolo, in which the volume of the voice changes repeatedly. The "parameter related to the variation over time" is, for example, the period, amplitude, or phase of the variation.

[0115] In a specific example (Aspect 4) of any one of Aspects 1 to 3, the voice conversion by the voice processing plug-in assigned to the first track is a conversion to a first voice quality, and the voice conversion by the voice processing plug-in assigned to the second track is a conversion to a second voice quality different from the first voice quality, and the adjustment conditions of the music parameters in the first voice signal are different from the adjustment conditions of the music parameters in the second voice signal. In the above aspects, the adjustment conditions of the music parameters are different between the first voice signal of the first voice quality and the second voice signal of the second voice quality. Therefore, it is possible to generate a first voice signal reflecting a pronunciation tendency specific to the first voice quality and a second voice signal reflecting a pronunciation tendency specific to the second voice quality.

[0116] The "music parameter adjustment condition" is a condition for adjusting the music parameters of each audio signal to make the music parameters different between the first audio signal and the second audio signal. For example, the direction (increase / decrease) of change in the music parameter or the amount of change is exemplified as the "music parameter adjustment condition". For example, when focusing on the sounding timing, whether the sounding timing is changed forward (past) or backward (future) on the time axis, or the amount of change in the sounding timing on the time axis, are exemplified as the "adjustment condition". Also, when focusing on the pitch of the voice, whether the pitch is raised or lowered, or the amount of change in the pitch are exemplified as the "adjustment condition".

[0117] In a specific example (Aspect 5) of any one of Aspects 1 to 4, the voice conversion by the voice processing plug-in is a process of converting the voice into a voice quality selected by the user from a plurality of voice qualities. According to the above aspect, voice signals of various voice qualities that reflect the user's intentions or preferences can be generated by the voice processing plug-in.

[0118] In a specific example (aspect 6) of aspect 5, the selection of one of the plurality of voice qualities and the voice quality conversion by the voice processing plug-in are performed in parallel with the playback of the converted voice signal. According to the above aspect, the voice quality of the reproduced voice can be changed in parallel with the playback of the converted voice signal.

[0119] In a specific example (aspect 7) of any one of aspects 1 to 6, the voice conversion by the voice processing plug-in includes a process of generating first control data representing the acoustic characteristics of the voice signal by processing the voice signal before voice conversion using a trained first generation model, a process of generating second control data by executing a process for making the music parameters different on the first control data, and a process of generating the voice signal after voice conversion by processing the second control data using a trained second generation model. According to the above aspects, a process of generating a time series of control data from the voice signal before voice conversion using the first generation model, and a process of generating a voice signal after voice conversion from the control data using the second generation model are sequentially executed. Therefore, it is possible to generate a voice that is auditorily natural, as compared with a configuration in which voice conversion is realized by, for example, voice processing that directly processes a voice signal. In addition, the music parameters differ between the first voice signal and the second voice signal as a result of processing the intermediate control data. Therefore, the process for making the music parameters different is simplified compared to a configuration in which the music parameters are made different by performing voice processing on the first voice signal and the second voice signal after voice quality conversion.

[0120] An audio processing system according to one embodiment (embodiment 8) of the present disclosure includes an assignment unit that assigns to each of a plurality of tracks in music production software an audio processing plug-in including voice quality conversion of an audio signal set on the track, and an adjustment unit that differentiates a first musical parameter related to a first audio signal after processing by the audio processing plug-in assigned to a first track among the plurality of tracks from a second musical parameter related to a second audio signal after processing by the audio processing plug-in assigned to a second track different from the first track.

[0121] A program according to one embodiment (embodiment 9) of the present disclosure causes a computer system to function as an assignment unit that assigns, to each of a plurality of tracks in music production software, an audio processing plug-in including voice quality conversion of an audio signal set on that track, and an adjustment unit that differentiates a first musical parameter related to a first audio signal after processing by the audio processing plug-in assigned to a first track among the plurality of tracks from a second musical parameter related to a second audio signal after processing by the audio processing plug-in assigned to a second track different from the first track.

[0122] The audio processing software according to one embodiment (embodiment 10) of the present disclosure includes an audio processing plug-in assigned to each of a plurality of tracks in the music production software, the audio processing plug-in including voice quality conversion of an audio signal set for the track, and an adjustment program for differentiating a first musical parameter related to a first audio signal after processing by the audio processing plug-in assigned to a first track among the plurality of tracks from a second musical parameter related to a second audio signal after processing by the audio processing plug-in assigned to a second track different from the first track. [Explanation of symbols]

[0123] 100...audio processing system, 11...control device, 12...storage device, 13...display device, 14...operation device, 15...sound emission device, 20...music production screen, 21...play button, 22...stop button, 23 (23a, 23b, 23c, 23d, 23e)...operators, 31...allocation unit, 32-1, 32-2...audio processing unit, 33...signal synthesis unit, 41...signal acquisition unit, 42...signal analysis unit, 43...adjustment unit, 44...voice quality conversion unit, 45...signal output unit.

Claims

1. assigning, to each of a plurality of tracks in the music production software, an audio processing plug-in including a voice conversion for an audio signal set for the track; A first music parameter related to a first audio signal after processing by an audio processing plug-in assigned to a first track among the plurality of tracks is made different from a second music parameter related to a second audio signal after processing by an audio processing plug-in assigned to a second track different from the first track. A method of audio processing implemented by a computer system.

2. The first and second musical parameters include a timing of sound generation.

2. The audio processing method of claim 1.

3. The first and second musical parameters include at least one of a pitch, a volume, and a parameter related to a temporal variation of an acoustic characteristic.

3. The audio processing method according to claim 1 or 2.

4. the voice conversion by the audio processing plug-in assigned to the first track is a conversion to a first voice quality; the voice conversion by the audio processing plug-in assigned to the second track is a conversion to a second voice quality different from the first voice quality; The adjustment condition of the music parameter in the first audio signal is different from the adjustment condition of the music parameter in the second audio signal.

2. The audio processing method of claim 1.

5. The voice conversion using the voice processing plug-in is a process of converting the voice into a voice quality selected by the user from among a plurality of voice qualities.

2. The audio processing method of claim 1.

6. The selection of one of the plurality of voice qualities and the voice quality conversion by the voice processing plug-in are performed in parallel with the reproduction of the converted voice signal. The audio processing method of claim 5.

7. The voice conversion using the voice processing plug-in is as follows: a process of generating first control data representing an acoustic feature of the speech signal before voice conversion by processing the speech signal before voice conversion using a trained first generative model; a process of generating second control data by executing a process for making the musical parameters different on the first control data; and processing the second control data using a trained second generation model to generate the converted voice signal.

2. The audio processing method of claim 1.

8. an allocation unit that allocates, to each of a plurality of tracks in the music production software, an audio processing plug-in including a voice conversion plug-in for converting a voice signal set to the track; an adjustment unit that makes a first music parameter related to a first audio signal after processing by an audio processing plug-in assigned to a first track among the plurality of tracks different from a second music parameter related to a second audio signal after processing by an audio processing plug-in assigned to a second track different from the first track; A voice processing system comprising:

9. an allocation unit that allocates, to each of a plurality of tracks in the music production software, an audio processing plug-in including a voice conversion plug-in for converting a voice signal set to the track; an adjustment unit that differentiates a first musical parameter related to a first audio signal after processing by an audio processing plug-in assigned to a first track among the plurality of tracks from a second musical parameter related to a second audio signal after processing by an audio processing plug-in assigned to a second track different from the first track; A program that causes a computer system to function as a

10. an audio processing plug-in that is assigned to each of a plurality of tracks in the music production software and includes a voice conversion for a voice signal set for the track; an adjustment program for making a first musical parameter related to a first audio signal after processing by an audio processing plug-in assigned to a first track among the plurality of tracks different from a second musical parameter related to a second audio signal after processing by an audio processing plug-in assigned to a second track different from the first track; Audio processing software including

Images