Acoustic model training system and method

Abstract

To enable data to be used for training an acoustic model to be selected from a plurality of pieces of training data, so as to facilitate various manners of training.SOLUTION: An acoustic model training system includes: a first device that is connectable to a network and used by a first user; and a server that is connectable to the network. The first device uploads a plurality of sound waveforms to the server under control of the first user, selects one or more sound waveforms as a first waveform set from the plurality of sound waveforms having already been updated or going to be uploaded from now, and transmits a first execution instruction for a first training job with respect to an acoustic model that generates an acoustic feature quantity to the server. The server starts executing the first training job using the selected first waveform set on the basis of the first execution instruction from the first device, and provides the first device with a trained acoustic model having been trained by the first training job.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] One embodiment of the present invention relates to a system and method for training an acoustic model. [Background technology]

[0002] There are known sound synthesis technologies that synthesize the voice of a specific singer and the performance sound of a specific instrument. In particular, in sound synthesis technologies that use machine learning (e.g., Patent Documents 1 and 2), a fully trained acoustic model is required to output a synthetic voice with natural pronunciation based on the specific voice and performance sound based on the musical score data and audio data input by a user. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-076843 A [Patent Document 2] International Publication No. 2022 / 080395 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in order to fully train an acoustic model, it is necessary to label a huge amount of speech and performance sounds with linguistic features, which requires a huge amount of time and money. Therefore, only well-funded companies can train acoustic models, and the types of acoustic models are limited.

[0005] One of the objectives of one embodiment of the present invention is to make it possible to easily perform various types of training by making it possible to select data to be used for training an acoustic model from a plurality of training data. [Means for solving the problem]

[0006] An acoustic model training system according to an embodiment of the present invention includes a first device connectable to a network and used by a first user, and a server connectable to the network. The first device uploads a plurality of sound waveforms to the server under the control of the first user, selects one or more sound waveforms as a first waveform set from the plurality of sound waveforms already uploaded or to be uploaded, and transmits a first execution instruction of a first training job for an acoustic model that generates acoustic features to the server. The server starts execution of the first training job using the selected first waveform set based on the first execution instruction from the first device, and provides the first device with a trained acoustic model trained by the first training job.

[0007] A method for training an acoustic model according to one embodiment of the present invention includes implementing, by one or more computers, the step of providing an interface to a first user that allows the user to select one or more sound waveforms from a plurality of pre-stored sound waveforms for executing a first training job for an acoustic model that generates acoustic features. Effect of the Invention

[0008] According to an embodiment of the present invention, by making it possible to select data to be used for training an acoustic model from a plurality of training data, it becomes possible to easily perform various types of training. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing the overall configuration of an acoustic model training system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing a configuration of a server in one embodiment of the present invention. [Diagram 3] FIG. 2 is a block diagram showing the concept of an acoustic model in one embodiment of the present invention. [Figure 4] FIG. 2 is a sequence diagram showing a method for training an acoustic model and a method for synthesizing speech in one embodiment of the present invention. [Diagram 5]FIG. 2 is a diagram showing an example of a GUI in a method for training an acoustic model in one embodiment of the present invention. [Figure 6] FIG. 2 is a sequence diagram showing a method for training an acoustic model and a method for synthesizing speech in one embodiment of the present invention. [Figure 7] FIG. 2 is a diagram showing an example of a GUI related to disclosure of acoustic model information and a preview request in one embodiment of the present invention. [Figure 8] FIG. 2 is a sequence diagram showing a method for training an acoustic model and a method for synthesizing speech in one embodiment of the present invention. [Figure 9] FIG. 13 is a diagram illustrating an example of a GUI for setting public information during training of an acoustic model in one embodiment of the present invention. [Figure 10] 3 is a flow chart illustrating a method for training an acoustic model in accordance with an embodiment of the present invention. [Figure 11] FIG. 2 is a sequence diagram showing a method for recording sound waveforms used for training an acoustic model in one embodiment of the present invention. [Figure 12] FIG. 2 is a diagram showing a data structure managed by a server in one embodiment of the present invention. [Figure 13] FIG. 2 illustrates data sent to a server in training an acoustic model in one embodiment of the present invention. [Figure 14] 3 is a flow chart illustrating a method for training an acoustic model in accordance with an embodiment of the present invention. [Figure 15] 1 is a flow chart illustrating a method for recommending songs suitable for training an acoustic model in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, a system and method for training an acoustic model according to an embodiment of the present invention will be described in detail with reference to the drawings. The embodiment described below is an example of a form for implementing the present invention, and the present invention is not limited to these embodiments. In the drawings referred to in this embodiment, the same parts or parts having similar functions are given the same or similar symbols (symbols with A, B, etc. added after the numbers), and repeated explanations of them may be omitted.

[0011] In the following embodiments, "musical score data" refers to data including information on the pitch and intensity of notes, information on the phonemes of notes, information on the duration of note pronunciation, and information on performance symbols. For example, musical score data is data indicating at least one of the musical score and lyrics of a piece of music. The musical score data may be data indicating the time sequence of notes that make up the piece of music, or may be data indicating the time sequence of language that makes up the piece of music.

[0012] A "sound waveform" is waveform data of a voice, and the sound source that produces the voice is identified by a sound source ID. For example, a sound waveform is at least one of waveform data of singing and waveform data of musical instrument sounds. For example, a sound waveform includes waveform data of a singer's singing voice and musical instrument sounds captured via an input device such as a microphone. A sound source ID identifies the timbre of the singer's singing or the timbre of the musical instrument sounds. Among sound waveforms, a sound waveform that is input to generate a synthetic sound waveform using an acoustic model is called a "sound waveform for synthesis," and a sound waveform used to train the acoustic model is called a "training sound waveform." When there is no need to distinguish between a synthesis sound waveform and a training sound waveform, they are collectively simply called a "sound waveform."

[0013] The "acoustic model" has an input of the score feature of the score data and an input of the acoustic feature of the sound waveform. As the acoustic model, for example, an acoustic model having a score encoder 111, an acoustic encoder 121, a switching unit 131, and an acoustic decoder 133 described in International Publication No. 2022 / 080395 is used. This acoustic model has a function of generating an acoustic feature of a target sound waveform having a timbre indicated by the sound source ID by processing the score feature of the input score data or the acoustic feature of the sound waveform and the sound source ID, and is a voice synthesis model used by a voice synthesis program for generating a new synthetic sound waveform. The voice synthesis program supplies the score feature generated from the score data of a certain piece of music and the sound source ID to the acoustic model, thereby obtaining the acoustic feature of the piece of music with the timbre indicated by the sound source ID, and converting the acoustic feature into a sound waveform. Alternatively, the speech synthesis program supplies acoustic features generated from the sound waveform of a certain piece of music and a sound source ID to an acoustic model to obtain new acoustic features of the piece of music in the timbre indicated by the sound source ID, and converts the new acoustic features into a sound waveform. A predetermined number of sound source IDs are prepared for each acoustic model. In other words, each acoustic model selectively generates acoustic features of the timbre indicated by the sound source ID from among the predetermined number of timbres.

[0014] An acoustic model is a generative model of a given architecture that uses machine learning, for example, a convolutional neural network (CNN) or a recurrent neural network (RNN). Acoustic features represent pronunciation characteristics in the frequency spectrum of the waveform of natural or synthetic sounds, and similar acoustic features mean that the timbre of the singing voice or performance sound and its time change are similar.

[0015] In training the acoustic model, the variables of the acoustic model are changed so that the acoustic model generates acoustic features similar to those of the referenced sound waveform. For example, the training program P2 described in International Publication No. 2022 / 080395, the score data D1 (training score data), and the learning acoustic data D2 (training sound waveform) are used for training. The variables of the acoustic model (score encoder, acoustic encoder, and acoustic decoder) are changed so that acoustic features of a synthetic sound of a plurality of timbres corresponding to a plurality of sound source IDs can be generated by basic training using waveforms of a plurality of voices corresponding to a plurality of sound source IDs. Furthermore, by supplementarily training the trained acoustic model using a sound waveform of another timbre corresponding to a new (unused) sound source ID, the acoustic model can generate acoustic features of the timbre indicated by the new sound source ID. Specifically, by further performing supplementary training on an acoustic model already trained with the sound waveforms of the voices of XXX (multiple people) using a new sound source ID and the sound waveforms of the voice of YYY (one person), the variables of the acoustic model (at least the acoustic decoder) are changed so that the acoustic model can generate acoustic features of YYY's voice. The unit of training of the acoustic model corresponding to the new sound source ID as described above is called a "training job." In other words, a training job means a series of training processes executed by a training program.

[0016] A "program" refers to an instruction or group of instructions executed by a processor in a computer having a processor and memory. A "computer" is a general term referring to the entity that executes a program. For example, when a program is executed by a server (or a client), the "computer" refers to the server (or the client). Also, when a "program" is executed by distributed processing between a server and a client, the "computer" includes both the server and the client. In this case, the "program" includes "a program executed by a server" and "a program executed by a client." Similarly, when a "program" is processed in a distributed manner among multiple servers, the "computer" includes the multiple servers, and the "program" includes each program executed by each server.

[0017] [1. First embodiment] [1-1. Overall system configuration] Fig. 1 is a diagram showing the overall configuration of an acoustic model training system in one embodiment of the present invention. As shown in Fig. 1, the acoustic model training system 10 includes a server 100, a communication terminal 200 (TM1), and a communication terminal 300 (TM2). The server 100 and the communication terminals 200, 300 can each be connected to a network 400. The communication terminals 200 and 300 can each communicate with the server 100 via the network 400. The communication terminal 200 may be referred to as a "first device." A user who uses the communication terminal 200 may be referred to as a "first user."

[0018] In this embodiment, the server 100 is a computer that functions as a speech synthesizer and trains an acoustic model. The server 100 includes a storage 110. In FIG. 1, a configuration in which the storage 110 is directly connected to the server 100 is illustrated as an example, but the configuration is not limited to this. For example, the storage 110 may be connected to a network 400 directly or via another computer, and data may be transmitted and received between the server 100 and the storage 110 via the network 400.

[0019] The communication terminal 200 is a terminal that selects a training sound waveform for training an acoustic model and transmits an instruction to execute the training to the server 100. The communication terminal 300 is a terminal different from the communication terminal 200 and is a terminal that can access the server 100. Although details will be described later, the communication terminal 300 is a terminal for viewing or listening to public information related to the acoustic model under training. The communication terminals 200 and 300 include, for example, mobile communication terminals such as smartphones or tablet terminals, or stationary communication terminals such as desktop personal computers.

[0020] The network 400 may be the Internet provided by a general World Wide Web (WWW) service, a Wide Area Network (WAN), or a Local Area Network (LAN) such as an in-house LAN.

[0021] [1-2. Server configuration used for speech synthesis] Fig. 2 is a block diagram showing the configuration of a server according to an embodiment of the present invention. As shown in Fig. 2, server 100 includes a control unit 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, a user interface (UI) 104, a communication interface 105, and storage 110. The speech synthesis technology of this embodiment is realized by the cooperation of the functional units of server 100.

[0022] The control unit 101 includes a central processing unit (CPU), a graphics processing unit (GPU), and storage devices such as registers and memories connected to the CPU and GPU. The control unit 101 executes programs temporarily stored in the memory using the CPU and GPU, and realizes each function provided in the server 100. Specifically, the control unit 101 performs calculation processing in response to various request signals from the communication terminal 200, and provides content data to the communication terminals 200 and 300.

[0023] The RAM 102 temporarily stores a control program required for arithmetic processing, an acoustic model (consisting of an architecture and variables), content data, etc. The RAM 102 is also used, for example, as a data buffer, and temporarily holds various data received from an external device such as the communication terminal 200 until the data is stored in the storage 110. The RAM 12 may be, for example, a general-purpose memory such as an SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory).

[0024] The ROM 103 stores various programs, various acoustic models, parameters, etc. for implementing the functions of the server 100. The programs, acoustic models, parameters, etc. stored in the ROM 103 are read out by the control unit 101 as necessary and executed or used.

[0025] The user interface 104 displays various display images such as a graphical user interface (GUI) on its display under the control of the control unit 101, and accepts input from the user of the server 100.

[0026] The communication interface 105 is an interface that is connected to the network 400 under the control of the control unit 101 and transmits and receives information to and from other communication devices such as the communication terminals 200 and 300 connected to the network 400 .

[0027] The storage 110 is a recording device (recording medium) capable of permanently retaining and rewriting information such as a non-volatile memory or a hard disk drive. The storage 110 stores information such as a program, an acoustic model, and parameters required for executing the program. As shown in FIG. 2, the storage 110 stores, for example, a voice synthesis program 111, a training job 112, musical score data 113, and a sound waveform 114. These programs and data can be those related to general voice synthesis, and for example, the voice synthesis program P1, training program P2, musical score data D1, and acoustic data D2 described in International Publication No. WO 2022 / 080395 may be used.

[0028] As described above, the voice synthesis program 111 is a program for generating a synthetic voice waveform from musical score data and a sound waveform. When the control unit 101 executes the voice synthesis program 111, the control unit 101 generates a synthetic voice waveform using the acoustic model 120. The synthetic voice waveform corresponds to the acoustic data D3 described in WO 2022 / 080395. The training program for the acoustic model 120 executed by the control unit 101 in the training job 112 is, for example, a program for training the encoder and the acoustic decoder described in WO 2022 / 080395. The musical score data is data that specifies a musical piece. The sound waveform is waveform data of a voice or a performance sound, for example, waveform data indicating a singer's singing voice or a musical instrument's performance sound.

[0029] [1-3. Functional configuration of the server used for speech synthesis] FIG. 3 is a block diagram showing the concept of an acoustic model in one embodiment of the present invention. As described above, the acoustic model 120 is a machine learning model used in the voice synthesis technology executed by the control unit 101 in FIG. 2 when the control unit 101 reads and executes the voice synthesis program 111. The acoustic model 120 generates an acoustic feature. The control unit 101 inputs the score feature 123 of the score data 113 of a desired piece of music or the acoustic feature 124 of the sound waveform 114 as an input signal to the acoustic model 120. The acoustic model 120 processes the sound source ID and the score feature 123 to generate an acoustic feature 129 of a synthetic sound of the piece of music. The control unit 101 synthesizes and outputs a synthetic sound waveform 130 of the piece of music sung by a singer specified by the sound source ID or played on an instrument based on the acoustic feature 129. Alternatively, the acoustic model 120 processes the sound source ID and the acoustic feature 124 to generate an acoustic feature 129 of a synthetic sound of the piece of music. Based on the acoustic feature quantity 129, the control unit 101 synthesizes and outputs a synthetic sound waveform 130 by converting the sound waveform of the song into the tone color of the singer's singing voice or the performance sound of an instrument specified by the sound source ID.

[0030] The acoustic model 120 is a generative model that uses machine learning, and is trained by the control unit 101 that is executing a training program (i.e., executing the training job 112). The control unit 101 trains the acoustic model 120 using a (new) sound source ID and a training sound waveform, and determines the variables of the acoustic model 120 (at least the acoustic decoder). Specifically, the control unit 101 generates training acoustic features from the training sound waveform, and when a new sound source ID and training acoustic features are input to the acoustic model 120, gradually and repeatedly changes the above variables so that the acoustic features that generate the synthetic sound waveform 130 approach the training acoustic features. The training sound waveform may be uploaded (transmitted) from the communication terminal 200 or the communication terminal 300 to the server 100, for example, and stored in the storage 110 as user data, or may be stored in the storage 110 in advance by the administrator of the server 100 as reference data. In the following description, storing in the storage 110 may be referred to as storing in the server 100.

[0031] [1-4. Voice synthesis method] FIG. 4 is a sequence diagram showing a method for training an acoustic model and a method for synthesizing speech in an embodiment of the present invention. In the method for training an acoustic model shown in FIG. 4, an example is shown in which the communication terminal 200 uploads a training sound waveform to the server 100. However, as described above, the training sound waveform may be stored in advance in the server 100 by other methods. The training job in the sequence shown in FIG. 4 may be referred to as a "first training job." Each step of the process TM1 on the communication terminal 200 side and each step of the process Server on the server 100 side are actually executed by the control unit of the communication terminal 200 and the control unit 101 of the server 100, respectively, but here, for simplicity of explanation, the communication terminal 200 and the server 100 are expressed as the executing entities of each step. Unless otherwise specified, the following explanation of the sequence diagram and the explanation of the flowchart are the same.

[0032] As shown in FIG. 4, first, the communication terminal 200 (first device) uploads (transmits) one or more training sound waveforms to the server 100 based on an instruction from the first user who has logged in to the account of the first user on the server 100 (step S401). The server 100 stores the training sound waveforms transmitted in S401 in the memory area of ​​the first user (step S411). The number of sound waveforms uploaded to the server 100 may be one or more, and the multiple sound waveforms may be stored in multiple folders in the memory area of ​​the first user. The above steps S401 and S411 are steps related to preparation for executing the following training job.

[0033] Next, the steps for executing a training job will be described below. The communication terminal 200 requests the server 100 to execute a training job (step S402). In response to the request of S402, the server 100 provides the communication terminal 200 with a graphical user interface (GUI) for selecting a sound waveform to be used for the training job from among the sound waveforms stored in advance (and sound waveforms to be stored) (step S412).

[0034] The communication terminal 200 displays the GUI provided in S412 on its display, and the first user uses the GUI to select one or more training sound waveforms from the multiple sound waveforms uploaded to the storage area (or a desired folder) as the waveform set 149 (see FIG. 5) (step S403). After the waveform set 149 (training sound waveforms) is selected in S403, the communication terminal 200 instructs the start of execution of a training job in response to an instruction from the first user (step S404).

[0035] Based on the instruction from the communication terminal 200 (first device) in S404, the server 100 starts executing the training job using the selected waveform set 149 (step S413). In other words, in S413, the training job is executed based on the instruction of the first user via the GUI provided in S412.

[0036] For training, not all of the waveforms in the selected waveform set 149 are used, but a preprocessed waveform set including only useful sections excluding silent sections and noise sections is used. In addition, an acoustic model 120 with an untrained acoustic decoder may be used as the acoustic model 120 to be trained (base acoustic model). However, the time and cost required for a training job can be reduced by selecting and using an acoustic model 120 including an acoustic decoder that has learned to generate acoustic features similar to those of the waveforms in the waveform set 149 from among multiple acoustic models 120 that have been basically trained. Whichever acoustic model 120 is selected, the score encoder and the acoustic encoder are both basically trained.

[0037] The base acoustic model may be determined by the server 100 based on the waveform set 149 selected by the first user. Alternatively, the first user may select one of a plurality of trained acoustic models as the base acoustic model and include designation data indicating the base acoustic model in the first execution instruction. A new unused sound source ID is used as the sound source ID (e.g., singer ID, instrument ID, etc.) to be supplied to the acoustic decoder. Here, the user does not necessarily need to know which sound source ID has been used as the new sound source ID. However, when performing voice synthesis using the trained model, the new sound source ID is automatically used.

[0038] In the training job, a unit training is repeated in which some short waveforms are extracted little by little from the preprocessed waveform set, and the extracted short waveforms are used to train the acoustic model (at least the acoustic decoder). In the unit training, the new sound source ID and the acoustic features of the short waveforms are input to the acoustic model 120, and the variables of the acoustic model are adjusted accordingly so that the difference between the acoustic features output by the acoustic model 120 and the input acoustic features becomes small. For example, the backpropagation method is used for adjusting the variables. By repeating the unit training, once the training using the preprocessed waveform set is completed, the quality of the acoustic features generated by the acoustic model 120 is evaluated, and if the quality does not reach a predetermined standard, the preprocessed waveform set is used to train the acoustic model again. If the quality of the acoustic features generated by the acoustic model 120 reaches a predetermined standard, the training job is completed, and the acoustic model 120 at that point becomes the trained acoustic model 120.

[0039] When the training job is completed in S413, the trained acoustic model 120 is established (step S414). This trained acoustic model 120 may be referred to as a "first acoustic model." The server 100 notifies the communication terminal 200 that the trained acoustic model 120 has been established (step S415). The above steps S403 to S415 are the training job for the acoustic model 120.

[0040] After the notification in S415, in response to an instruction from the first user, the communication terminal 200 transmits an instruction for voice synthesis, including the score data of the desired music piece, to the server 100 (step S405). In response to this, the server 100 executes a voice synthesis program and executes voice synthesis using the trained acoustic model 120 completed in S414 based on the score data (step S416). The synthetic voice waveform 130 generated in S416 is transmitted to the communication terminal 200 (step S417). In this voice synthesis, the new sound source ID is used.

[0041] In combination, S416 and S417 can be said to provide the trained acoustic model 120 (voice synthesis function) trained by the training job to the communication terminal 200 (first device) or the first user. The execution of the voice synthesis program in step S416 may be performed by the communication terminal 200 instead of the server 100. In this case, the server 100 transmits the trained acoustic model 120 to the communication terminal 200, and the communication terminal 200 uses the received trained acoustic model 120 to execute voice synthesis processing based on the sheet music data of the desired music piece with the new sound source ID, and acquires the synthetic voice waveform 130.

[0042] In this embodiment, the training sound waveform is uploaded in S401 before the execution of the training job is requested in S402, but the present invention is not limited to this configuration. For example, the training sound waveform may be uploaded after the execution of the training job is instructed in S404. In this case, in S403, one or more sound waveforms may be selected as the waveform set 149 from a plurality of sound waveforms (including sound waveforms that have not been uploaded) stored in the communication terminal 200, and the sound waveforms that have not been uploaded from among the selected sound waveforms may be uploaded in response to an instruction to execute the training job.

[0043] [1-5.GUI140] Here, an example of the GUI provided in S412 will be described. FIG. 5 is a diagram showing an example of a first GUI in the acoustic model training method in one embodiment of the present invention. The GUI 140 shown in FIG. 5 is displayed on a display included in the user interface of the communication terminal 200. As shown in FIG. 5, the GUI 140 displays sound waveform A, sound waveform B, ..., and sound waveform Z (for example, sound waveforms uploaded to a specific folder) as candidates for training sound waveforms. Check boxes 141, 142, ..., and 143 are displayed next to each sound waveform. The sound waveform A, sound waveform B, ..., and sound waveform Z displayed as candidates for training sound waveforms as described above are, for example, sound waveforms related to the singing voice of the same person, and may each have a different song or singing style. The sound waveforms may be various sounds played by the same instrument.

[0044] In other words, in S412, the server 100 provides the communication terminal 200 with a GUI that allows the first user to select one or more sound waveforms as a waveform set 149 from a plurality of pre-stored sound waveforms (and sound waveforms to be stored) for executing a training job for the acoustic model 120.

[0045] In S403 above, the first user of the communication terminal 200 checks the check boxes 141, 142, ..., 143 shown in Fig. 5 to select training sound waveforms. Fig. 5 shows an example in which the check boxes 141 and 142 are checked as training sound waveforms, and sound waveforms A and B are selected as waveform set 149. One or more waveforms may be selected as waveform set 149.

[0046] In the above S404, when the execute button 144 is pressed with the checkboxes 141 and 142 selected, the communication terminal 200 executes the instruction of the training job of S404. In response to the instruction of the training job, the server 100 starts training the acoustic model 120 using the waveform set 149 consisting of the sound waveform A and the sound waveform B. Pressing the execute button 144 includes clicking or tapping the execute button 144.

[0047] As described above, the acoustic model training system 10 according to the present embodiment selects one or more sound waveforms from a plurality of sound waveforms (and sound waveforms to be stored) stored in advance in the storage 110, and executes a training job for the acoustic model 120 using the selected sound waveforms as training sound waveforms. With the above configuration, the first user of the communication terminal 200 obtains a desired acoustic model 120 by training an untrained or trained acoustic model 120. Note that the sound waveforms may be uploaded to the server 100 after the selection of the waveform set 149 or the instruction to execute a training job. In other words, the sound waveforms used in the training job may be uploaded from the communication terminal 200 to the server 100 at any time before the training job is started. Furthermore, if the acoustic decoder is auxiliary training of a trained acoustic model, the trained acoustic model 120 can be obtained in a short time and at a low cost compared to the conventional acoustic model 120.

[0048] [2. Second embodiment] An acoustic model training system 10A according to the second embodiment will be described with reference to Figs. 6 and 7. The overall configuration of the acoustic model training system 10A and the block diagram relating to the server are the same as those of the acoustic model training system 10 according to the first embodiment, and therefore the description will be omitted. In the following description, the description of the same configuration as in the first embodiment will be omitted, and differences from the first embodiment will be mainly described. In the following description, when describing the same configuration as in the first embodiment, the alphabet "A" will be added after the reference numerals shown in these figures with reference to Figs. 1 to 5.

[0049] [2-1. Voice synthesis method] Fig. 6 is a sequence diagram showing an acoustic model training method and a speech synthesis method according to an embodiment of the present invention. In the acoustic model training method shown in Fig. 6, a configuration is described in which information indicating the progress of a training job is made public to a third party from when execution of the training job is started at the instruction of a user until a trained acoustic model is completed. Steps prior to step S601 in Fig. 6 are the same as steps S401 to S403 in Fig. 4, and therefore description thereof is omitted. S601 in Fig. 6 is the same as S404 in Fig. 4. In the following description, a user who uses communication terminal 300A and corresponds to the third party may be referred to as a "second user".

[0050] Based on the execution instruction from the first user from the communication terminal 200A in S601, the server 100A starts execution of a training job of the base acoustic model using a new sound source ID and the selected waveform set 149A (step S611). At the completion of the training job, a trained acoustic model 120A trained with this waveform set 149A is obtained as a result. When the training job is started in S611, the server 100A notifies the communication terminal 200A that the training job has been started, and inquires the communication terminal 200A about whether or not to disclose status information indicating the status of the training job to a third party, that is, whether or not to allow viewing by a third party (step S612). If the first user issues a disclosure instruction to disclose status information indicating the status of the training job in response to the inquiry in S612, the communication terminal 200A transmits the disclosure instruction to the server 100A (step S602). If the first user does not issue a disclosure instruction, the communication terminal 200A does not transmit a disclosure instruction. This status information is transmitted to communication terminal 200A regardless of whether or not there is an instruction to make it public, and is displayed on the display device thereof for viewing by the first user.

[0051] In S602, based on the disclosure instruction by the first user as described above, the server 100A discloses to the communication terminal 300A the status information indicating the status of the training job of the first user that was started in S611 (step S613). This allows a third party to view the status information displayed on the display of the communication terminal 300A.

[0052] In addition, if the first user has agreed in advance to make the status information indicating the status of the training job public and has issued a disclosure instruction, steps S612 and S602 can be omitted. In other words, the status information indicating the status of the training job of the first user may be made public to the second user based on the disclosure instruction issued in advance by the first user.

[0053] Steps S615 to S618 after S622 are similar to steps S414 to S417 in FIG. 4, so the description will be omitted.

[0054] 6 illustrates a configuration in which a communication terminal 300A different from the communication terminal 200A that issued an instruction to execute a training job is the subject that executes the preview request, but the present invention is not limited to this configuration. For example, the communication terminal 200A (first user) that issued an instruction to execute a training job may execute a preview request in order to check the progress of the training job himself. For example, by communication terminal 200A making a preview request, the training job can be ended at a timing when the first user is satisfied with the synthesized voice waveform for preview, even if the progress has not reached 100%.

[0055] [2-2.GUI150A] Here, an example of the GUI provided in S613 will be described. Fig. 7 is a diagram showing an example of a GUI related to disclosure of acoustic model information and a preview request in an embodiment of the present invention. The GUI 150A shown in Fig. 7 is displayed on the display of the communication terminals 200A and 300A.

[0056] 7, the GUI 150A displays an item 151A indicating a progress level according to the state information, an item 152A indicating detailed information, and a preview button 157A for requesting preview. In this embodiment, the item 151A indicating the progress level indicates the progress level of the training job of the acoustic model 120A. However, the item 151A may be an item other than the degree of completion, such as the elapsed time when the expected completion rate is 100% and the degree of change in the variables of the acoustic model 120A.

[0057] Item 151A is a progress bar that displays the progress of the training job in percentage. In item 151A, the current state indicated by the progress may be the current training amount relative to the total training amount estimated at the start of the training job, or the current training amount relative to the total training amount estimated from the change in the variables of the acoustic model 120A during the execution of the training job. In other words, the state of the training job changes over time, and the server 100A provides and displays the progress indicating the change over time of the state of the training job as item 151A to the communication terminal. Since the state of the training job changes over time, the server 100A repeatedly updates the state information indicating the state of the training job when the information changes or at regular intervals, and repeatedly provides it to the communication terminals 200A and 300A.

[0058] In the present embodiment, a configuration has been exemplified in which status information indicating the status of a training job is repeatedly provided to the communication terminals 200A and 300A in real time, but the present invention is not limited to this configuration. For example, the status information may be provided only once to each of the communication terminals 200A and 300A. Alternatively, the status information may be displayed on the communication terminal 300A (second device) based on a disclosure request made by a second user using the communication terminal 300A.

[0059] 7, a progress bar is displayed as the item 151A indicating the degree of progress, but the present invention is not limited to this. For example, the degree of progress may be displayed as a numerical value in the form of a percentage.

[0060] Item 152A is information showing details of the training job. In FIG. 7, an acoustic model name 153A, a training sound waveform 154A, a completion forecast 155A, and a training executor 156A are displayed as examples of detailed information of item 152A. The acoustic model name 153A is a name set by the first user. For example, "voice X→Y" means that the pre-training acoustic model 120A (base acoustic model) that synthesizes the voice of X (one or more singers X, or one or more musical instruments X) is changed to a trained acoustic model 120A that synthesizes the voice of Y (a new singer Y or musical instrument Y) by the training job being executed. The training sound waveform 154A indicates a sound waveform used for training the acoustic model 120A in the training job being executed. The example in FIG. 7 means that the sound waveform B is used for the acoustic model 120A. The completion forecast 155A indicates the date and time when the progress of the training job being executed is expected to reach 100%. The training executor 156A indicates the name of the user who executed the training job being executed. The user name may be an account name or a nickname. In FIG. 7, the training executor 156A is "U1". U1 may be the same as or different from the singer or performer related to Y.

[0061] The preview button 157A is a button for executing a preview request, which will be described later. For example, in FIG. 6, after the information disclosure in S613, the second user presses the preview button 157A, and the communication terminal 300A requests the server 100A to preview the synthetic voice (step S621). When the preview request is executed in S621, the server 100A executes a preview speech synthesis using the acoustic model 120A of the progress level at the time when the preview request is executed, using the new sound source ID, and provides a synthetic voice waveform for preview (step S614). By providing the synthetic voice waveform for preview, the communication terminal 300A can preview the synthetic voice generated by the acoustic model 120A at the above time (step S622). Naturally, this preview can also be performed by the communication terminal 200A.

[0062] Training jobs are executed in batches, with each batch being a group of processes. When the above preview request is executed, if the acoustic model 120A is in the middle of one batch process, the preview synthetic sound waveform generated by the acoustic model 120A obtained in the immediately preceding batch process may be provided, or after that point, when the ongoing batch process is completed, the preview synthetic sound waveform generated by the obtained acoustic model 120A may be provided. In other words, based on the preview request from the communication terminals 200A and 300A, the server 100A provides the first and second users with the preview synthetic sound waveform generated by the acoustic model 120A according to the timing of the preview request.

[0063] As described above, according to the acoustic model training system 10A of the present embodiment, the second user of the communication terminal 300A can view the process in which the acoustic model 120A is trained and established by the training job. Alternatively, the first user of the communication terminal 200A can end the training job at a satisfactory timing even if the progress does not reach 100%, as described above.

[0064] [3. Third embodiment] An acoustic model training system 10B according to the third embodiment will be described with reference to Figs. 8 and 9. The overall configuration of the acoustic model training system 10B and the block diagram relating to the server are the same as those of the acoustic model training system 10 according to the first embodiment, and therefore the description will be omitted. In the following description, the description of the same configuration as in the first embodiment will be omitted, and differences from the first embodiment will be mainly described. In the following description, when describing the same configuration as in the first embodiment, reference will be made to Figs. 1 to 5, and the alphabet "B" will be added after the reference numerals shown in these figures.

[0065] [3-1. Voice synthesis method] Fig. 8 is a sequence diagram showing an acoustic model training method and a speech synthesis method in one embodiment of the present invention. In the acoustic model training method shown in Fig. 8, a first training job and a second training job are executed in parallel, and a configuration in which state information regarding each training job is selectively made public to a third party will be described. Steps prior to step S801 in Fig. 8 are the same as S401 to S403 in Fig. 4, and therefore description thereof will be omitted. S801 in Fig. 8 is the same as S404 in Fig. 4.

[0066] Based on the first execution instruction from the communication terminal 200B in S801, the server 100B executes a first training job of the first base acoustic model using a new sound source ID and the first waveform set selected by the first user (step S811). When the first training job is started in S811, the server 100B notifies the communication terminal 200B that the first training job has been started, and inquires of the communication terminal 200B whether or not the first status information related to the first training job should be made public to a third party (step S812). In this embodiment, the above "third party" corresponds to the second user. In response to the inquiry in S812, the communication terminal 200B transmits a disclosure instruction to the server 100B to disclose the first status information (step S802).

[0067] In S802, based on the first disclosure instruction by the first user as described above, the server 100B discloses the first status information regarding the first training job executed in S811 to the communication terminal 300B (second user) (step S813). If the first user does not issue the first disclosure instruction, the server 100B does not disclose the first status information to the second user.

[0068] Next, based on the second execution instruction from the communication terminal 200B in S803, the server 100B executes a second training job of the second base acoustic model using a new sound source ID and a second waveform set selected by the first user (step S814). The first training job and the second training job are executed in parallel by S811 and S814. The first base acoustic model and the second base acoustic model are independent of each other, and there is no correlation between the sound source IDs used by the two. For example, when n training jobs are processed in parallel, this is realized by starting n virtual machines. The second waveform set used in the second training job is different from the first waveform set used in the first training job, but the training program of the second training job is the same as the training program of the first training job. When the first training job is completed, the first trained acoustic model trained with the first waveform set is obtained as a result, and when the second training job is completed, the second trained acoustic model trained with the second waveform set is obtained as a result.

[0069] The method of executing the second training job is similar to the method of executing the first training job, except that the second training job uses a second waveform set, which is one or more sound waveforms selected by the first user from a plurality of sound waveforms previously stored (and sound waveforms to be stored).

[0070] In S814, when the second training job is started, the server 100B notifies the communication terminal 200B that the second training job has been started, and inquires of the communication terminal 200B whether or not the second status information regarding the second training job should be made public (step S815). In response to this inquiry, the communication terminal 200B transmits a second disclosure instruction to the server 100B to disclose the second status information regarding the second training job (step S804). The server 100B that has received the second disclosure instruction discloses the second status information regarding the second training job executed in S814 to the communication terminal 300B (second user) (step S816). If the first user has not issued the second disclosure instruction, the server 100B does not disclose the second status information to the second user.

[0071] If the first user has agreed in advance to the disclosure of the status information regarding the first training job or the second training job and has issued a disclosure instruction, steps S812, S802, S815, and S804 can be omitted. In other words, the status information regarding the first training job or the second training job may be made public to the second user based on the disclosure instruction issued in advance by the first user.

[0072] Steps S831 to S821 after S816 are basically the same as steps S621 to S618 in FIG. 6, but are executed separately for each of the first training job and the second training job.

[0073] [3-2.GUI160B] Here, an example of a GUI provided to the first user in S815 will be described. Fig. 9 is a diagram showing an example of a public setting GUI when setting public information during training of an acoustic model in one embodiment of the present invention. GUI 160B shown in Fig. 9 is displayed on the display of communication terminal 200B of the first user.

[0074] As shown in FIG. 9, GUI 160B is a screen for setting what information is to be made public when the status information of the training job is made public. In this embodiment, the public setting item 161B includes a first training job item 162B and a second training job item 167B. In the first training job 162B, items of an acoustic model name 163B, a training sound waveform 164B, a completion forecast 165B, and a training person 166B are displayed as examples of detailed settings. In the second training job 167B, items of an acoustic model name 168B, a training sound waveform 169B, a completion forecast 170B, and a training person 171B are displayed as examples of detailed settings. The above items are the same as the items shown in FIG. 7, and therefore description thereof will be omitted.

[0075] In GUI 160B of FIG. 9, items selected by the user are displayed as "black rectangles (■)", and items not selected by the user are displayed as "white rectangles (□)". When an item of first training job 162B is selected by the first user, all detailed items related to the first training job are automatically selected. In this case, all items related to the first training job are made public. When an item of second training job 167B is not selected, the first user can individually select detailed items related to the second training job. In the case of FIG. 9, only the items of acoustic model name 168B and training sound waveform 169B are selected. In this case, only the selected detailed items of the second training job are made public. The first communication terminal transmits a first disclosure instruction to the server 100B for the range of information selected by the first user as the disclosure target among the first status information of the first training job (S802 and S804), and transmits a second disclosure instruction for the range of information selected by the first user as the disclosure target among the second status information of the second training job (S804). That is, the server 100B individually and selectively discloses (provides to the communication terminal 300B) at least one of the first status information and the second status information to the second user based on the disclosure instruction from the first user. For items of the first training job and the second training job for which no disclosure instruction has been received, the corresponding status information is not disclosed to the second user.

[0076] Note that a GUI similar to the above is provided in S812, but in that GUI, only items related to the first training job 162B are displayed.

[0077] The public button 172B is a button for instructing disclosure of information on the acoustic model under training. In S804 in Fig. 8, when the first user presses the public button 172B, a public instruction for the items to be publicly disclosed selected by the user from the status information of the first training job and the second training job is transmitted from the communication terminal 200B to the server 100B, and the status information of the items to be publicly disclosed is made public to a third party in the same format as in Fig. 7 (step S816).

[0078] As described above, according to the acoustic model training system 10B of the present embodiment, the first user can individually make a plurality of training jobs that the first user has started public to a third party. In addition, the first user can freely set which items to make public and which items not to make public for each detailed item of the training job.

[0079] [4. Fourth embodiment] An acoustic model training system 10C according to the fourth embodiment will be described with reference to FIG. 10. The overall configuration of the acoustic model training system 10C and the block diagram relating to the server are the same as those of the acoustic model training system 10 according to the first embodiment, and therefore the description will be omitted. In the following description, the description of the same configuration as in the first embodiment will be omitted, and differences from the first embodiment will be mainly described. In the following description, when describing the same configuration as in the first embodiment, reference will be made to FIGS. 1 to 5, and the alphabet "C" will be added after the reference numerals shown in these figures.

[0080] [4-1. Voice synthesis method] Fig. 10 is a flowchart showing a method for training an acoustic model in one embodiment of the present invention. In the method for training an acoustic model shown in Fig. 10, a training job instructed by a user is executed on the condition that the user has paid the charge. Fig. 10 explains the operations performed from the instruction of a training job in S404 in Fig. 4 to the execution of the training job in S413. Steps S1001 and S1004 in Fig. 10 are the same as S404 and S413 in Fig. 4, respectively.

[0081] As shown in FIG. 10, in S1001, the communication terminal 200C transmits an instruction to execute a training job (first execution instruction) to the server 100C. Then, the server 100C that receives the execution instruction charges the first user who instructed the execution of the training job, and notifies the communication terminal 200C of information related to the charge (step S1002). After the notification, the server 100C determines whether the communication terminal 200C has paid the charge to the operator of the server 100C (step S1003). When the communication terminal 200C has paid the charge ("Yes" in S1003), the server 100C executes the instructed training job on the base acoustic model using the selected waveform set within the range of the charge (step S1004). On the other hand, if the communication terminal 200C does not make the payment ("No" in S1003), the training job is not executed by the server 100C, and an error (non-execution of the training job) is notified to the communication terminal 200C (step S1005). The server 100C executes the billing process of S1002 each time the control unit of the server 100C executes a training job for a unit time (S1004), and upon receiving payment from the first user (S1003), may execute a training job for the next unit time for the acoustic model under training (S1004).

[0082] As described above, according to the acoustic model training system 10C according to this embodiment, the first user can cause the server 100C to execute training jobs that correspond to the amount paid.

[0083] [5. Fifth Embodiment] An acoustic model training system 10D according to the fifth embodiment will be described with reference to Figs. 11 to 14. The overall configuration of the acoustic model training system 10D and the block diagram relating to the server are the same as those of the acoustic model training system 10 according to the first embodiment, and therefore the description will be omitted. In the following description, the description of the same configuration as in the first embodiment will be omitted, and differences from the first embodiment will be mainly described. In the following description, when describing the same configuration as in the first embodiment, the alphabet "D" will be added after the reference numerals shown in these figures with reference to Figs. 1 to 5.

[0084] [5-1. Voice synthesis method] Fig. 11 is a sequence diagram showing a method for recording a sound waveform used for training an acoustic model in one embodiment of the present invention. In the recording method shown in Fig. 11, a configuration is described in which a training sound waveform is recorded in a recording space such as a karaoke booth and uploaded to a server. The recording space is a real space. In the following description, a rental space is exemplified as the recording space.

[0085] The karaoke server 500D shown in FIG. 11 is, for example, a server or computer that manages the rental of karaoke boxes and karaoke booths. The karaoke server 500D manages a space ID that identifies one of a plurality of rental spaces, such as karaoke boxes and karaoke booths, provided in one store, and an availability that indicates whether each rental space is available. The rental space may be a completely closed space, such as a karaoke box, or a space that is partially open to the outside, such as a karaoke booth. Each rental space is equipped with a karaoke machine that has a recording function and a communication function with the karaoke server 500D. The karaoke server 500D can be connected to the network 400D and can communicate with the server 100D via the network 400D. In this embodiment, the server 100D performs a reservation service for the rental space for the karaoke server 500D. However, the details will be described later, but the present invention is not limited to this configuration.

[0086] First, the communication terminal 200D logs in to the acoustic model training service provided by the server 100D (step S1101). In S1101, the communication terminal 200D transmits account information (e.g., a user ID and a password) input by a first user who uses the service to the server 100D. The server 100D performs user authentication based on the account information received from the communication terminal 200D, and approves the first user's login to the account with that user ID (step S1111). The user authentication may be performed by an external authentication server instead of the server 100D.

[0087] The communication terminal 200D requests the reservation of a rental space with a desired space ID at a desired date and time including the use of the service, using the user ID logged in at S1111 (step S1102). When the server 100D receives the reservation request at S1102, it checks the usage status or availability of the rental space with the space ID at the date and time with the karaoke server 500D (step S1112). If the rental space is available, the karaoke server 500D makes a reservation (step S1121), and transmits reservation completion information to the server 100D indicating that the reservation of the rental space with the space ID at the date and time has been completed. If the first user has specified prepayment in the reservation request, the rental fee and the service usage fee are charged at step S1121. The service usage fee is a fee for a basic training job using the waveform recorded in the rental space, which is executed after the rental space is used. The communication terminal 200D may make a reservation request for a rental space to the karaoke server, and in response to the reservation request, the karaoke server 500D that made the reservation may transmit reservation completion information including the user ID and space ID related to the reservation to the server 100D.

[0088] When the server 100D receives the reservation completion information from the karaoke server 500D (step S1113), it links the space ID related to the reservation completion information with the user ID of the first user (step S1114). Then, it notifies the communication terminal 200D that the reservation is completed (step S1115). The reservation completion notification may be sent from the karaoke server 500D to the communication terminal 200D.

[0089] When the communication terminal 200D receives the reservation completion notification, the communication terminal 200D displays to the first user that the reservation has been completed, as well as information identifying the reserved rental space and date and time. The information identifying the rental space is, for example, a room number of a karaoke booth identified by a space ID. When the first user moves to the reserved rental space on the reserved date and time and operates the karaoke equipment provided in the rental space to select a desired song, the accompaniment to the song is played in the rental space. The first user executes a recording start instruction and a recording end instruction using the karaoke equipment. In response to these instructions, the karaoke server 500D records the first user's singing voice or the sound of the instrument played (step S1122).

[0090] When the usage time of the rental space ends (recording is completed), the karaoke server 500D (rental company) charges the first user for the rental space and the training job if the usage fee has not been paid in advance, and the first user pays the usage fee at the terminal of the karaoke server 500D. Since the usage fee is set together with the rental fee, the usage fee for the training job may be discounted from the fee charged in S1002. The first user selects a sound waveform to be uploaded to the server 100D from the sound waveforms (waveform data) that have been recorded, and further, when the usage fee for the training job has been paid, selects a waveform set to be used for the training job from the sound waveforms to be uploaded. The karaoke server 500D uploads the selected sound waveform and the space ID where the recording was performed to the memory area of ​​the first user specified by the user ID of the first user of the server 100D (step S1123).

[0091] The server 100D stores the uploaded sound waveform and the space ID in the storage area of ​​the first user in a linked manner (step S1116). The number of sound waveforms uploaded and stored in the server 100D may be one or more.

[0092] In S1114, the space ID and the user ID of the first user are linked, and in S1116, the uploaded sound waveform and the space ID are linked. Therefore, as shown in FIG. 12, the server 100D links and stores the user ID 180D of the first user, the space ID 181D, and the uploaded sound waveform 182D. FIG. 12 is an example of data managed by the server in one embodiment of the present invention. The user ID 180D is the user ID of the account logged in at S1111 in FIG. 11, and each data in FIG. 13 described later is stored in a storage area corresponding to the user ID. The space ID 181D is the space ID of the space where the sound was recorded at S1122 in FIG. 11. The sound waveform 182D is the sound waveform recorded at S1122 in FIG. 11 and transmitted to the server 100D at S1123.

[0093] The server 100D identifies the user ID of the first user who uploaded the sound waveform from the storage area where the sound waveform was uploaded in S1123 (step S1117). After that, based on an instruction from the first user, the server 100D executes a training job of the base acoustic model using the new sound source ID and the uploaded sound waveform (step S1118).

[0094] Here, the data uploaded from the karaoke server 500D to the server 100D in S1123 will be described with reference to Fig. 13. In the description of Fig. 11, a configuration in which only the sound waveforms representing the singing voice or performance sound of the first user are uploaded to the server 100D in S1123 has been exemplified, but this configuration is not limiting. For example, in the case of singing voice, as shown in Fig. 13, pitch data 503D representing the sounds constituting the guide melody of the music provided to the rental space by the karaoke device and text data 502D representing the lyrics of the music may be uploaded to the server 100D together with the sound waveforms 501D. In the case of performance sound, the text data 502D is not uploaded.

[0095] The step of uploading the data recorded in S1122 to the server 100D in S1123 by the karaoke server 500D will be described with reference to FIG. 14. In the description of FIG. 11, a configuration in which the sound waveform recorded in S1122 is uploaded to the server 100D in S1123 without going through any particular steps has been exemplified, but the present invention is not limited to this configuration. For example, as shown in FIG. 14, the first user may determine whether or not to upload the sound waveform after playing back the voice data related to the recorded sound waveform. In the example of FIG. 14, the karaoke device or the communication terminal 200D is used to inquire of the first user whether or not to play back the recorded sound waveform, whether or not to upload the sound waveform, whether or not to re-record, and whether or not to end the operation. These four inquiries may be displayed in order on one GUI, or may be displayed in parallel on the GUI as a play button, an upload button, a re-record button, and an end button.

[0096] After the recording of the audio data is completed in S1122 of FIG. 11, as shown in FIG. 14, the karaoke server 500D judges whether or not the first user has issued a playback instruction (step S1401). If a playback instruction has been issued in S1401 ("Yes" in S1401), the karaoke server 500D uses the karaoke equipment to play the audio data recorded in S1122 of FIG. 11 in the rental space where the recording was made (step S1402). During the playback, the audio data alone may be played back, or the audio data may be played back together with a guide melody. After the playback is performed in S1402, the process returns to step S1401. If there is no playback instruction in S1401 ("No" in S1401), the process proceeds to the next step without executing the playback in S1402.

[0097] Next, it is determined whether or not to upload the voice data recorded in S1122 of Fig. 11 (step S1403). For example, the karaoke server 500D provides the first user with a GUI for selecting whether or not to upload the recorded voice data, and determines whether or not to upload according to the selection made by the first user.

[0098] If it is determined in S1403 that uploading is necessary ("Yes" in S1403), the upload in S1123 in Fig. 11 is executed, and the above operation ends. On the other hand, if there is no instruction to execute uploading in S1403 ("No" in S1403), a determination is made as to whether re-recording is necessary (step S1404). For example, the karaoke server 500D provides the first user with a GUI for selecting whether or not to re-record, and determines whether or not to re-record according to the selection made by the first user.

[0099] If it is determined in S1404 that re-recording is necessary ("Yes" in S1404), the karaoke server 500D re-records in the same manner as S1122 in FIG. 11 (step S1405). When the re-recording in S1405 is finished, the presence or absence of a playback instruction is determined again in S1401. If there is no instruction to start re-recording in S1404 ("No" in S1404), it is determined whether or not to end the operation (step S1406). If it is determined in S1406 that the operation can be ended ("Yes" in S1406), the above operation ends. On the other hand, if there is no instruction to end the operation in S1406 ("No" in S1406), the process returns to step S1401. If there is no instruction to play in S1401, an instruction to execute upload in S1403, an instruction to start re-recording in S1404, or an instruction to end in S1406, the karaoke server 500D repeatedly executes these determination steps.

[0100] In the present embodiment, the server 100D performs the reservation service for the rental space on behalf of the karaoke server 500D, but the present invention is not limited to this configuration. For example, the karaoke server 500D may perform the reservation service for the rental space. In this case, the server 100D and the karaoke server 500D share the first account information of the first user. Furthermore, the server 100D links the space ID and the sound waveform received from the karaoke server 500D to the user ID (first account information) of the first user and stores them. The subsequent steps are the same as those from S1122 onward in FIG. 11.

[0101] The recording start instruction and recording end instruction in S1122 in FIG. 11 may be executed by the start and end of the song, or may be executed by an arbitrary operation of the first user. In other words, based on the recording instruction of the first user, only audio data for a specified period of the playback period of the song may be recorded. The recording start instruction and recording end instruction may be executed using a karaoke machine, or may be executed using the communication terminal 200D. In other words, the recording in S1122 may be executed only for at least a part of the playback period of the song. In other words, as shown in FIG. 13, the server 100D may receive from the karaoke server 500D pitch data 503D indicating the notes of the part of the song sung or played by the first user provided in the rental space and text data 502D indicating the lyrics of the song, together with sound waveforms 501D that are audio data recording singing during at least a part of the playback period of the song. The server 100D then stores the sound waveforms 501D of the singing or playing sounds as training sound waveforms, linked to the musical score data.

[0102] As described above, according to the acoustic model training system 10D of this embodiment, voice data can be recorded using a karaoke booth or the like and uploaded to the server 100D, thereby reducing the effort required for the first user to prepare an environment for recording voice data.

[0103] [6. Sixth Embodiment] An acoustic model training system 10E according to the sixth embodiment will be described with reference to FIG. 15. The overall configuration of the acoustic model training system 10E and the block diagram relating to the server are the same as those of the acoustic model training system 10 according to the first embodiment, and therefore the description will be omitted. In the following description, the description of the same configuration as in the first embodiment will be omitted, and differences from the first embodiment will be mainly described. In the following description, when describing the same configuration as in the first embodiment, reference will be made to FIG. 1 to FIG. 5, and the alphabet "E" will be added after the reference numerals shown in these figures.

[0104] [6-1. Voice synthesis method] Fig. 15 is a flowchart showing a method for recommending a piece of music suitable for training a target acoustic model in one embodiment of the present invention. The recommendation method shown in Fig. 15 describes a configuration for recommending to a first user a piece of music suitable for a sound waveform based on all or a part of a sound waveform previously stored in the server 100E as a training sound waveform, or a waveform set selected by the user. The communication terminal 100E has received information from the first user in advance indicating the range of use of the acoustic model with respect to pitch or acoustic feature quantity that the first user expects.

[0105] First, the server 100E analyzes the training sound waveforms stored in advance or the selected waveform set (step S1501). The training sound waveforms to be analyzed are not all of the stored training sound waveforms, but are part of the training sound waveforms of a specific sound source (a specific singer or a specific instrument). For example, a folder for each singer or instrument may be provided in the memory area of ​​the first user of the server 100E, and the training sound waveforms may be stored separately in the folders for the corresponding singer or instrument, and the analysis may be performed individually for the sound waveforms stored in each folder. The waveform set is a set of sound waveforms of a specific singer or a specific instrument selected by the first user to train an acoustic model of a specific singer or a specific instrument. The analysis is performed, for example, based on the pitch or acoustic features of the sound waveform. Furthermore, if the music piece of the analyzed sound waveform is known, the singing skill or playing skill can be judged in terms of pitch, timbre, dynamics, etc. by comparing the sound waveform with the score data of the sung or played sound of the music piece. Alternatively, the analysis may determine a singing style, a playing style, a singing range, or a playing range.

[0106] A singing style is a way of singing, and a performance style is a way of performing. Specifically, singing styles include neutral, vibrato, husky, fly, and growl. Performance styles include neutral, vibrato, pizzicato, spiccato, flageolet, and tremolo for bowed string instruments, and neutral, position, legato, slide, and slap / mute for plucked string instruments. For a clarinet, neutral, staccato, vibrato, and trill are examples. For example, the above vibrato refers to a singing style or performance style that uses vibrato a lot. The pitch, volume, timbre, and their dynamic behaviors in singing or performance vary depending on the style as a whole. In a training job, the server 100E may train the base acoustic model 120E while inputting a singing style or performance style obtained by analyzing the waveform set in addition to a new timbre ID and waveform set.

[0107] The singing range and performance range of the training sound waveforms are determined from the distribution of pitches in a plurality of sound waveforms of a specific singer and a specific instrument, and indicate the range of the sound waveforms of that singer or instrument.

[0108] When the range of pitch data and acoustic features to be used for the timbre of a specific sound source is not covered, the server 100E determines that the prepared training sound waveforms are not sufficient for training the acoustic model. By performing the analysis of S1501, the server 100E detects a range that has no or few sound waveforms among the entire range in which the timbre of a specific sound source is to be used. Then, the server 100E identifies one or more songs to be recommended to the first user in order to supplement the range with insufficient data (step S1502). Then, the server 100E provides information indicating the songs identified in S1502 to the communication terminal 200E (first user), and the communication terminal 200E displays the received information on its display.

[0109] As described above, according to the acoustic model training system 10E of this embodiment, if the sound waveform prepared as the training sound waveform does not cover the intended range of use, the first user is notified of this, so that the first user can prepare a training sound waveform that covers the intended range of use.

[0110] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present invention. The embodiments can be combined with each other as long as no technical contradiction occurs. [Explanation of symbols]

[0111] 10: Acoustic model training system, 100: Server, 101: Control unit, 102: RAM, 103: ROM, 104: User interface, 105: Communication interface, 110: Storage, 111: Speech synthesis program, 112: Training job, 113: Music score data, 114: Sound waveform, 120: Acoustic model, 130: Synthesized sound waveform, 140: GUI, 141, 142, 143: Checkbox, 144: Execute button, 150A: GUI, 151A, 152A: Items indicating progress, 153A: Acoustic model name, 154A: Training sound waveform, 155A: Completion expectation, 156A: Training executor, 157A: Preview button, 160B: GUI, 161B: Public setting item, 162B: First training job, 163B, 168B: Acoustic model name, 164B, 169B: Training sound waveform, 165B, 170B: Completion expectation, 166B, 171B: Training executor, 167B: Second training job, 172B: Publish button, 180D: Account information, 182D: Sound waveform, 200, 300: Communication terminal, 400: Network, 411: Step, 500D: Karaoke server, 501D: Sound waveform, 502D: Text data, 503D: Sound pitch data

Claims

1. A first device used by the first user that can connect to a network, Includes a server that can connect to the aforementioned network, The first device selects one or more sound waveforms from a plurality of sound waveforms as a first waveform set, An acoustic model training system comprising the server or the first device, which trains an acoustic model that generates acoustic features using the selected first waveform set, and generates a trained acoustic model.

2. The server starts executing a first training job for the acoustic model using the selected first waveform set based on a first execution instruction from the first device. The acoustic model training system according to claim 1, wherein the trained acoustic model trained by the first training job is provided to the first device.

3. The plurality of sound waveforms are stored in the server or the first device, The acoustic model training system according to claim 1, wherein the first device selects the first waveform set from the plurality of sound waveforms under the control of the first user.

4. The plurality of sound waveforms are already stored in the server or the first device, The acoustic model training system according to claim 1, wherein the first device selects the first waveform set from the plurality of sound waveforms under the control of the first user.

5. The acoustic model training system according to claim 1, wherein each sound waveform included in the first waveform set is a sound waveform already stored in the server, or a sound waveform uploaded from the first device to the server.

6. The training system for an acoustic model according to claim 1, wherein the trained acoustic model generates acoustic features for generating sound waveforms.

7. A method for training an acoustic model implemented by one or more computers, which provides a first user with an interface that allows them to select one or more sound waveforms to be used in a first training job for an acoustic model that generates acoustic features from multiple pre-stored sound waveforms.

8. Using the interface, the first user selects one or more waveforms to receive as a first waveform set. Based on the first execution instruction from the first user via the interface, the execution of the first training job is started using the first waveform set. The training method according to claim 7, further comprising providing the acoustic model trained by the first training job to the first user as the first acoustic model.

9. The training method according to claim 8, further comprising providing a second user, different from the first user, with first state information indicating the state of the first training job, based on a first disclosure instruction from the first user.

10. The first status information is displayed on the first device used by the first user. The training method according to claim 9, further comprising displaying the first state information on a second device used by the second user based on the first disclosure instruction.

11. The state of the first training job changes over time. The training method according to claim 9, wherein the first state information displayed on the second device used by the second user is repeatedly updated.

12. The training method according to claim 9, wherein the progress of the status of the first training job is displayed as the first status information.

13. The training method according to claim 9, wherein, based on a disclosure request by the second user, the first state information at the timing of the disclosure request is displayed on a second device used by the second user.

14. Using the interface described above, the first user receives one or more waveforms newly selected as a second waveform set. The process further includes initiating the execution of a second training job using the second waveform set based on a second execution instruction from the first user, The training method according to claim 8, wherein the first training job and the second training job are executed in parallel.

15. The training method according to claim 14, further comprising providing at least one of the first state information relating to the first training job and the second state information relating to the second training job to a second device of a second user different from the first user, based on a disclosure instruction from the first user.

16. In response to the first execution instruction from the first user, the first user is charged, The training method according to claim 7, wherein the execution of the first training job is started when payment for the aforementioned charge is confirmed.

17. Receive a spatial ID that identifies the real space, The training method according to claim 7, wherein the account information of the first user for the service providing the training method is linked to the spatial ID.

18. The training method according to claim 17, wherein a charge is made to the first user having the account information linked to the spatial ID.

19. The musical score data representing the sounds that make up the song, reproduced in the aforementioned real space, is received together with audio data containing recordings of singing or playing sounds during at least a portion of the playback period of the song. The training method according to claim 17, wherein the aforementioned audio data is stored in advance as a sound waveform and linked to the aforementioned musical score data.

20. The training method according to claim 19, wherein, based on the recording instruction of the first user, the audio data for a specified period within the playback period is recorded.

21. Based on the playback instruction from the first user, the audio data is played back in the physical space. The training method according to claim 19, further comprising asking the first user whether or not to save the audio data played back by the playback instruction as one of the plurality of pre-stored sound waveforms provided to the first user.

22. By analyzing the previously saved sound waveforms, Based on the analysis results obtained from the above analysis, the songs to be recommended to the first user are identified, The training method according to claim 7, further comprising providing the first user with information indicating the identified musical piece.

Images