Information processing system, information processing method, and information processing program

Abstract

An information processing system according to the present disclosure comprises: a reception unit that receives, from a user, subjective evaluation information relating to speech evaluation, and designation of a speaking style; a generation unit that, on the basis of the subjective evaluation information and the designation of the speaking style, converts a low-dimensional embedding vector corresponding to the speaking style adjusted by the user to generate a high-dimensional speaking style embedding vector; and a speech synthesis unit that synthesizes speech from the speaking style embedding vector and text information defining speech to be synthesized.

Description

Information Processing System, Information Processing Method, and Information Processing Program

[0001] The present disclosure relates to an information processing system, an information processing method, and an information processing program.

[0002] Conventionally, a voice synthesis technique having a function of generating a synthesized voice by designating a speech style such as emotion has been known. For example, in the prior art, in order to generate a synthesized voice according to a user's preference, it has been proposed to compress a speech style represented by a high-dimensional vector into a low-dimensional vector and learn the user's preference for voice quality (Patent Document 2).

[0003] Japanese Patent Application Laid-Open No. 9-325773 Japanese Patent Application Laid-Open No. 2017-167273

[0004] W. Yuxuan, S. Daisy, et al., “Style Tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis.” International conference on machine learning. PMLR, 2018.

[0005] However, in the prior art, since the speech style cannot be easily adjusted at the time of generating the synthesized voice, the usability for adjusting the speech style cannot be improved.

[0006] Therefore, the present disclosure proposes an information processing system, an information processing method, and an information processing program that can improve the usability for adjusting the speech style.

[0007] To solve the above problems, one form of information processing system according to this disclosure includes: a receiving unit that receives subjective evaluation information relating to the evaluation of speech and a specification of speech style from a user; a generating unit that generates a high-dimensional speech style embedding vector by converting a low-dimensional embedding vector corresponding to the speech style adjusted by the user based on the subjective evaluation information and the specification of speech style; and a speech synthesis unit that synthesizes speech from the speech style embedding vector and text information defining the speech to be synthesized.

[0008] Figure 1 shows an overview of the process for generating synthesized speech. Figure 2 shows an overview of the process for generating speech style embedding vectors. Figure 3 shows an overview of the learning process for adjustment parameters performed by the information processing device. Figure 4 shows an example of the configuration of the information processing device according to the embodiment. Figure 5 shows an example of generating low-dimensional embedding vectors. Figure 6 shows an example of presenting speech style embedding vectors in a mapping space. Figure 7 shows an example of displaying a face image on the mapping space. Figure 8 shows an example of a UI for the user to adjust the speech style. Figure 9 shows an example of receiving impression words to be evaluated for speech. Figure 10 shows an example of receiving evaluation of sample speech. Figure 11 shows an example of multiple regression analysis. Figure 12 shows an example of clustering results. Figure 13 shows an example of learning between a speech synthesis model and a style encoder. Figure 14 shows an example of compressed representation learning. Figure 15 shows an example of singular value decomposition. Figure 16 shows an example of learning compression and reconstruction. Figure 17 shows an example of transmitted information. Figure 18 shows an example of the configuration of the information processing system according to the embodiment. Figure 19 is a diagram showing an example configuration of an information processing system according to the embodiment. Figure 20 is a flowchart showing an example of a processing procedure for generating synthesized speech according to the embodiment. Figure 21 is a flowchart showing an example of a processing procedure for generating a speech style embedding vector according to the embodiment. Figure 22 is a flowchart showing an example of a processing procedure for generating adjustment parameters according to the embodiment. Figure 23 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device.

[0009] Embodiments of this disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals to avoid redundant descriptions.

[0010] This disclosure will be described in the following order of items: 1. Prior Art 2. Embodiments 2-1. Overview of the Information Processing System According to the Embodiment 2-2. Configuration of the Information Processing Device According to the Embodiment 2-3. Example Configuration of the Inference Process According to the Embodiment 2-3-1. Example Configuration of the Speech Synthesis Process According to the Embodiment 2-4. Example Configuration of the Learning Process According to the Embodiment 2-4-1. Generation of Adjustment Parameters According to the Embodiment 2-4-2. Learning of the Speech Synthesis Model and Style Encoder According to the Embodiment 2-4-3. Learning of the Compressed Representation According to the Embodiment 2-5. Example Configuration of the Information Processing System According to the Embodiment 3. Information Processing Procedure According to the Embodiment 4. Other Embodiments 5. Effects of the Information Processing Device According to the Disclosure 6. Hardware Configuration

[0011] (1. Conventional Technology) In recent years, speech synthesis technologies that allow users to specify their speech style have become known. However, with such speech synthesis technologies, users cannot easily and intuitively adjust the speech style. Furthermore, conventional speech synthesis technologies have made it difficult to fine-tune the speech style. Speech style refers to characteristics such as pitch.

[0012] For example, speech style adjustments can be made using text or parameters. However, fine-tuning speech style using text is difficult. Similarly, adjusting speech style using parameters is difficult because the number of parameters the user needs to adjust becomes enormous due to the diverse ways in which speech styles can be expressed.

[0013] Furthermore, since speech style adjustments are based on the user's subjective impression, it is difficult to obtain common parameter factors shared among users. Therefore, conventional speech style adjustments often do not correlate well with the user's subjective impression.

[0014] Therefore, this disclosure proposes an information processing device that improves the usability of adjusting speech style.

[0015] (2. Embodiments) (2-1. Overview of the Information Processing System According to the Embodiment) First, an overview of the processing of the present disclosure according to the embodiment will be described using Figures 1, 2, and 3. As an example of the processing of the present disclosure, the processing performed by an information processing device 100 (not shown) will be described. The information processing device 100 is a computer that generates synthesized speech based on a specified speech style.

[0016] In this disclosure, "speech style" refers to characteristics related to speech. For example, speech style refers to characteristics related to speech such as speaker identity, age, emotional expression, intonation, and dialect. Speech style also includes characteristics that arise due to the influence of the surrounding environment and recording equipment, such as noise in the recorded speech and the frequency characteristics of the microphone. Specifically, characteristics of the surrounding environment and recording equipment refer to noise that occurs in the recorded speech and characteristics such as the absence of output of certain frequencies, such as low frequencies.

[0017] Figure 1 is a diagram illustrating the overview of the process for generating synthesized speech. As shown in Figure 1, the information processing device 100 receives the content to be output as speech (utterance content) from the user as text 11 in order to output the synthesized speech desired by the user (step S11). The information processing device 100 also has a speech synthesis model 5 that outputs synthesized speech 17 from the text 11 that defines the speech to be synthesized. The speech synthesis model 5 is input as an intermediate feature, which is a high-dimensional speech style embedding vector 24 that defines the speech style.

[0018] The information processing device 100 converts the input text (string) 11 into phonemes (step S12). For example, the information processing device 100 uses g2p (Grapheme to phoneme) 12 to convert the input text into phonemes. g2p 12 is a trained model that has been trained to convert text into phonemes.

[0019] The information processing device 100 extracts features from the phoneme-converted text using the text encoder 13. The text encoder 13 is a pre-trained model that has been trained to extract features from the phoneme-converted text.

[0020] The information processing device 100 adds or concatenates a vector that defines the speech style (hereinafter also referred to as the speech style embedding vector 24), which will be described later, to the feature quantities extracted from the text 11 that has been converted into phonemes (step S13).

[0021] The information processing device 100 uses the decoder 14 to generate a Mel spectrogram 15 from the features of the text into which the speech style embedding vectors 24 are added or concatenated (step S14). The Mel spectrogram 15 is a spectrogram that takes into account human sound perception. Note that the generated spectrogram is not limited to a Mel spectrogram; other spectrograms that show features of time and frequency may also be used. The decoder 14 is a trained model that has been trained to convert the features of the text into which the speech style embedding vectors 24 are added or concatenated into a spectrogram.

[0022] The information processing device 100 inputs the generated Mel spectrogram 15 to the waveform generation model 16 (step S15). The waveform generation model 16 is a trained model that has been trained to reconstruct the spectrogram into waveform data. When waveform data is output from the waveform generation model 16, the information processing device 100 outputs synthesized speech 17 corresponding to the waveform data (step S16).

[0023] Next, we will explain an example of accepting user specification (adjustment) of speech style. Figure 2 is a diagram showing an overview of the process for generating the speech style embedding vector 24. As described above, the information processing device 100 adds or concatenates the speech style embedding vector 24 to the feature quantities (intermediate features) output as intermediate representations in the process of outputting synthesized speech 17 from text input.

[0024] Furthermore, the extraction of speech style embedding vectors 24 and the conditioning of the speech synthesis model 5 may be carried out at multiple levels. For example, the extraction of speech style embedding vectors 24 and the conditioning of the speech synthesis model 5 may be carried out by first extracting embedding vectors related to "speaker identity" and then extracting embedding vectors related to "emotion".

[0025] The information processing device 100 generates a speech style embedding vector 24 to be added to or concatenated with the feature quantities output as intermediate representations of the speech synthesis model 5. The speech style embedding vector 24 is a high-dimensional vector (for example, 512 dimensions or 1024 dimensions) to represent various variations in speech style. However, adjusting each element of a high-dimensional vector is time-consuming and difficult. Therefore, the information processing device 100 accepts speech style adjustments using a low-dimensional vector.

[0026] The following describes the process for adjusting speech style using low-dimensional vectors. First, the information processing device 100 receives element values ​​21 of any number of impression words from the user. The elements are impression words that define speech style, such as speaker identity, age, emotional expression, intonation, and dialect. Element values ​​21 are encoded values ​​of the impression words.

[0027] The information processing device 100 accepts one or more impression words from the user. For example, the information processing device 100 accepts "anger" and "sadness" as emotional expressions.

[0028] Next, the information processing device 100 vectorizes the element values ​​21 using adjustment parameters (step S21). After receiving adjustments to the element values ​​21 of impression words specified by the user using adjustment parameters, the information processing device 100 vectorizes the adjusted element values ​​21, subjective evaluation information, etc., to generate a low-dimensional embedding vector.

[0029] Specifically, the information processing device 100 accepts adjustments to the element values ​​21 for "anger" and "sadness" from the user by manipulating adjustment parameters. The adjustment parameters are parameters of a trained model that accepts adjustments to the element values ​​21 and is trained to generate low-dimensional embedding vectors based on the element values ​​21 and subjective evaluation information. The adjustment parameters are obtained from subjective evaluation information received from the user, as described later.

[0030] The information processing device 100 inputs the low-dimensional embedding vector 22, which defines the speech style, into a reconstructor 23 that reconstructs it into a high-dimensional vector (step S22). The reconstructor 23 is a trained model that has been trained to convert low-dimensional vectors into high-dimensional vectors.

[0031] The information processing device 100 outputs a speech style embedding vector 24, which is a high-dimensional vector, from the low-dimensional embedding vector 22 input to the reconstructor 23 (step S23). The information processing device 100 adds or concatenates the speech style embedding vector 24 to the feature quantities output as intermediate representations (step S24).

[0032] Thus, the information processing device 100 can change the speech style using a low-dimensional embedding vector 22 according to each element of the input speech style embedding vector 24. However, the low-dimensional embedding vector 22 is not learned by explicitly assigning labels to speech styles during training. Therefore, in this disclosure, adjustment parameters based on subjective information are generated so that the adjustment can be interpreted and the speech style can be adjusted in accordance with the user's intentions.

[0033] Therefore, in this disclosure, the information processing device 100 generates adjustment parameters used to adjust the speech style before speech synthesis, separate from the speech synthesis described in Figures 1 and 2. The information processing device 100 has a learning stage in which it receives user feedback on the speech and learns adjustment parameters to adjust the speech style based on the user's subjectivity. Based on the feedback results, the information processing device 100 generates adjustment parameters that correspond to the user's subjectivity.

[0034] Adjustment parameters are parameters used to define a speech style based on the user's subjective perception. For example, adjustment parameters define the speech style based on evaluations of impression words specified by the user when the adjustment parameters are generated. Adjustment parameters are configured to allow adjustment of the intensity of the speech style. Adjusting the intensity of the speech style refers to adjusting how much of the speech style is reflected in the synthesized speech.

[0035] Figure 3 is a diagram illustrating the overview of the learning process for adjustment parameters performed by the information processing device 100. As shown in Figure 3, the information processing device 100 receives input of impression words 31 that define the speech style (step S31). The input impression words 31 are impression words that are subjectively evaluated by the user.

[0036] Next, the information processing device 100 receives input of an evaluation text 32 corresponding to the audio output as a sample audio (step S32). The sample audio is the audio subject to the user's subjective evaluation.

[0037] Next, the information processing device 100 extracts high-dimensional embedding vectors 34 from the speech dataset 33 (step S33). The speech dataset 33 is a dataset that covers a wide range of speech styles. After extracting the high-dimensional embedding vectors 34, the information processing device 100 may convert them into low-dimensional vectors using a compressor, which will be described later.

[0038] For example, the information processing device 100 clusters the extracted high-dimensional embedding vectors 34. The information processing device 100 also clusters the low-dimensional vectors compressed by the compressor. From each cluster separated by clustering, the information processing device 100 selects a high-dimensional embedding vector that represents the cluster, and selects the audio corresponding to that representative embedding vector as representative data 35 (step S34).

[0039] The information processing device 100 creates a presentation pattern 36 that presents the selected representative data 35 to the user (step S35). The information processing device 100 presents a group of sample voices 37 to the user from the selected representative data, the presentation pattern 36, and the evaluation text 32 (step S36).

[0040] The information processing device 100 receives a subjective evaluation response 38 from the user of the presented sample audio group 37 (step S37). For example, the information processing device 100 receives an evaluation of each of the presented sample audio based on impression words 31 specified by the user. Specifically, if the impression words 31 specified by the user are "anger" and "sadness," the information processing device 100 receives a subjective evaluation of the sample audio on a 10-point scale for "anger" and "sadness," respectively. Note that the method of subjective evaluation is not limited to receiving scores.

[0041] Multiple information processing devices 100 repeatedly present a group of sample voices 37 and receive subjective evaluations of the samples from the user (step S38). After receiving subjective evaluations from the user a predetermined number of times, the information processing devices 100 calculate a subjective evaluation value 39 (step S39). The subjective evaluation value 39 includes the impression words 31 specified by the user and the evaluation results associated with the impression words of the voice samples of the representative data in each cluster.

[0042] Here, the predetermined number of times may be any number set by the user, or it may be determined by a default setting or the like. The information processing device 100 generates adjustment parameters 40 from the calculated subjective evaluation values ​​39 corresponding to the representative data 35 (step S40). For example, the speech style defined by the adjustment parameters is defined by the subjective evaluation values ​​by the user.

[0043] The information processing device 100 can generate synthesized speech 17 according to the speech style desired by the user by receiving adjustments from the user using the generated adjustment parameters 40. In addition, the information processing device 100 can accept intuitive adjustments to the speech style from the user using parameters based on the user's subjective evaluation.

[0044] (2-2. Configuration of the Information Processing Apparatus According to the Embodiment) Next, the configuration of the information processing apparatus 100 according to the embodiment will be described with reference to FIG. 4. FIG. 4 is a diagram showing a configuration example of the information processing apparatus 100 according to the embodiment.

[0045] As shown in FIG. 4, the information processing apparatus 100 includes a communication unit 110, a storage unit 120, and a control unit 130. Note that the information processing apparatus 100 may include an input unit (e.g., a touch panel) that receives various operations from a user who operates the information processing apparatus 100 and a display unit (e.g., a liquid crystal display) that displays various information.

[0046] The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 110 is connected to the network N (the Internet, NFC (Near field communication), Bluetooth (registered trademark), etc.) by wire or wirelessly, and transmits and receives information to and from the terminal device 10 and the like via the network N.

[0047] The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 4, the storage unit 120 includes a model storage unit 121.

[0048] The storage unit 120 stores data used in the present embodiment. For example, the storage unit 120 stores a voice data set and answers to subjective evaluations received from the user. Note that the storage unit 120 may store a voice data set received from the user.

[0049] The model storage unit 121 stores the learned models used in the present embodiment. For example, the model storage unit 121 stores a voice synthesis model and a style encoder.

[0050] The control unit 130 is implemented, for example, by a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing a program (for example, the information processing program according to this disclosure) stored inside the information processing device 100 using RAM or the like as a working area. The control unit 130 is also a controller and may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0051] As shown in Figure 4, the control unit 130 includes a reception unit 131, a generation unit 132, a speech synthesis unit 133, a clustering unit 134, a compression unit 135, a display control unit 136, and a learning unit 137, and realizes or executes the information processing functions and operations described below.

[0052] First, as an example of the configuration of the control unit 130, an example configuration relating to the process of speech synthesis is shown. Next, as an example of the configuration of the control unit 130, an example configuration relating to the learning process, which includes the generation of adjustment parameters 40 for speech synthesis, is shown.

[0053] (2-3. Example of the configuration of the inference processing according to the embodiment) (2-3-1. Example of the configuration of the speech synthesis processing according to the embodiment) The reception unit 131 receives text information that defines the speech to be synthesized (spoken speech). For example, the reception unit 131 receives sentences or words as text information. Specifically, the reception unit 131 receives the text information "Hello. My name is XX." as the speech to be synthesized.

[0054] The reception unit 131 receives subjective evaluation information regarding the evaluation of the voice and the specification of the speaking style from the user. Subjective evaluation information refers to the user's specification of impression words 31 for the voice to be evaluated (sample voice) and the evaluation result of those impression words 31. Subjective evaluation information will be explained in the learning process described later.

[0055] For example, the reception unit 131 accepts the specification of one or more impression words 31 that specify a speech style, and the adjustment of the specified impression words 31 using adjustment parameters 40. Specifying a speech style means specifying the speech style of the synthesized voice that the user desires.

[0056] Specifically, the reception unit 131 accepts impression words 31 that are included in the impression words 31 received during the process of generating the adjustment parameters 40, as impression words 31 that define the speech style of the synthesized voice. For example, the reception unit 131 accepts "anger" and "sadness" as impression words 31.

[0057] For example, the adjustment parameter 40 includes a slider bar for adjusting the intensity of the impression word 31. The reception unit 131 accepts the adjustment of intensity using the slider bar for each impression word 31. To explain using the example of the impression words 31 "anger" and "sadness" mentioned above, the reception unit 131 accepts the user's operation to adjust the intensity of "anger" and "sadness" in the synthesized voice using the slider bar. The reception unit 131 may accept adjustments using the slider bar for each impression word 31 individually, or it may accept adjustments for multiple impression words 31 at once. Note that the adjustment of the intensity of the impression word 31 is not limited to a slider bar, but may also be done using a selector switch or the like.

[0058] The generation unit 132 converts the low-dimensional embedding vector corresponding to the speech style adjusted by the user, based on the subjective evaluation information and the specified speech style, to generate a high-dimensional speech style embedding vector.

[0059] The adjustment parameters 40 are generated based on subjective evaluation information, but the details will be described later. The generation unit 132 uses the adjustment parameters 40 to vectorize the element values ​​of the adjusted impression words 31, and generates a low-dimensional embedding vector.

[0060] The generation unit 132 may generate a low-dimensional embedding vector from the adjusted element values ​​of multiple impression words 31 (elements) that have been simultaneously adjusted (operated) by the user. Alternatively, the generation unit 132 may generate a low-dimensional embedding vector from the adjusted element values ​​of each of the multiple impression words 31 (elements) that have been directly adjusted (operated) by the user.

[0061] For example, if the generation unit 132 receives multiple impression words 31 of any number from the user, it concatenates vectors representing the adjusted values ​​of each impression word 31 to generate a low-dimensional embedding vector. The adjustment parameters 40 are, as mentioned above, parameters of a trained model that has been trained to accept adjustments to the impression words 31 received from the user and generate a low-dimensional embedding vector.

[0062] Figure 5 shows an example of generating a low-dimensional embedding vector. In Figure 5, an example is shown in which adjustments to multiple impression words 31 are accepted simultaneously and a low-dimensional embedding vector is generated. As shown in Figure 5, the reception unit 131 accepts adjustments to multiple impression words 31 by operating a slider bar 51 which is configured as an adjustment parameter.

[0063] The generation unit 132 generates a low-dimensional embedding vector from the element values ​​obtained by quantifying the adjusted impression words 31. The generation unit 132 generates a low-dimensional embedding vector 54 corresponding to the speech style by multiplying the adjustment value received by operating the slider bar 51, the subjective evaluation value 52 determined based on subjective evaluation information, and the unit vector 53. The slider bar 51, the subjective evaluation value 52, and the unit vector 53 are included in the configuration of the adjustment parameters.

[0064]

[0065] For example, the generation of the low-dimensional embedding vector 54 is expressed by the above equation. In equation (1), the low-dimensional embedding vector 54 can be obtained by simultaneously manipulating each of the k elements Xk of the low-dimensional embedding vector 54 (k ≤ m) by the same magnification S using a weight coefficient βk determined based on subjective evaluation information. Note that K = 1, ... k is the index of the dimension, and ek is the unit vector in the direction of the k-th dimension.

[0066] Next, the generation unit 132 uses a trained reconstructor to convert the low-dimensional embedding vector 54 into a high-dimensional speech style embedding vector.

[0067] The speech synthesis unit 133 synthesizes speech from speech style embedding vectors and text information defining the speech to be synthesized. For example, the speech synthesis unit 133 uses a speech synthesis model trained to synthesize speech from text information defining the speech to be synthesized (spoken speech) to synthesize speech.

[0068] Here, the speech synthesis unit 133 synthesizes speech by adding or concatenating feature quantities that define the speech style to the feature quantities of the text information extracted by the speech synthesis model in order to define the speech style.

[0069] For example, the speech synthesis model is a statistical model using a Hidden Markov Model, or a model utilizing a neural network. The model utilizing a neural network is not limited to a model that generates speech from text via a spectrogram (for example, the speech synthesis model in Figure 1), but may also be a model that directly generates speech from text. When the speech synthesis unit 133 generates speech via a spectrogram, it uses a waveform generation model to reconstruct the speech waveform using the time and frequency of the spectrogram as features, and then synthesizes the speech. Note that an existing trained model may be used as the waveform generation model.

[0070] For example, a spectrogram can be anything that includes time and frequency information. Specifically, a spectrogram may be a spectrogram showing time, frequency, and signal strength, a Mel spectrogram (for example, the speech synthesis model in Figure 1), or a log-Mel spectrogram where the frequency is shown logarithmically.

[0071] Next, we will explain speech styles in more detail. The following explanation will describe the process of visualizing speech styles. For example, the visualization of speech styles is intended to allow users to intuitively interact with speech styles.

[0072] The clustering unit 134 clusters the speech style embedding vectors into multiple clusters. For example, the clustering unit 134 clusters the speech style embedding vectors into multiple groups (hereinafter also referred to as clusters). The speech style embedding vectors to be clustered may be high-dimensional speech style embedding vectors, or they may be compressed, as described later. Note that existing methods such as the K-means method may be used for clustering. Details of the clustering will be described later and are therefore omitted here.

[0073] The compression unit 135 compresses high-dimensional speech style embedding vectors into low-dimensional embedding vectors. For example, the compression unit 135 compresses speech style embedding vectors into two or three dimensions using a trained compressor, which will be described later. Note that the compression of speech style embedding vectors is not limited to two or three dimensions.

[0074] Furthermore, the compression unit 135 can normalize the three-dimensionally compressed embedding vectors using a norm and convert them into unit vectors in order to plot the variables of the numerical vectors that constitute the speech style embedding vectors on a spherical surface, which will be described later. The norm represents the distance with respect to the vector space.

[0075] The display control unit 136 plots the numerical vector variables constituting the speech style embedding vectors separated by clustering on a mapping space in the UI (User Interface). For example, the display control unit 136 presents a map plotting the numerical vector variables constituting the speech style embedding vectors compressed into two or three dimensions on a two-dimensional plane or in three-dimensional space.

[0076] In the following description, the plane or space on which the speech style embedding vectors are plotted, such as a two-dimensional plane or a three-dimensional space, is referred to as the mapping space. The mapping space is not limited to a two-dimensional plane or a three-dimensional space, but may also be on a cylindrical surface or a spherical surface. Furthermore, the display control unit 136 may plot the numerical vector variables constituting the speech style embedding vectors within the cylindrical surface or spherical surface.

[0077] The clustering unit 134 clusters the variables of the numerical vectors that constitute the speech style embedding vector. The display control unit 136 may display the data of each clustered group in different colors.

[0078] Each value plotted in the mapping space corresponds to the element value 21 of the speech style embedding vector. In other words, values ​​that are close in distance in the mapping space represent speeches with similar speech styles.

[0079] Figure 6 shows an example of presenting speech style embedding vectors in mapping space. As shown in Figure 6, the display control unit 136 presents the numerical vector variables that constitute the speech style embedding vectors plotted on a two-dimensional plane 61. The display control unit 136 also presents the numerical vector variables plotted on a three-dimensional space 62. The display control unit 136 presents the numerical vector variables plotted on a cylinder 63. The display control unit 136 presents the numerical vector variables plotted on a sphere 64.

[0080] For example, the display control unit 136 may present the height (Z-axis) of a numerical vector variable plotted on the cylinder 63 as a characteristic of speech style, such as pitch. Alternatively, the display control unit 136 may present the numerical vector variable plotted on the sphere 64 as a characteristic of speech style that circulates around the sphere.

[0081] Note that while coordinate axis memory is shown in each mapping space in Figure 6, it is not necessary to show it. For example, the display control unit 136 may show normalized embedding vectors plotted in the mapping space so that the display range of each coordinate axis is from -1 to 1.

[0082] For example, the display control unit 136 presents a face image or a 3D model in the UI for at least one of the variables plotted on the mapping space. For example, the display control unit 136 presents a 3D model that shows the characteristics of the speech styles of each group divided by clustering. The display control unit 136 presents a person or character as the 3D model. Specifically, the display control unit 136 presents a 3D model at the center position of the coordinates of each cluster in order to show the characteristics of each cluster.

[0083] Furthermore, the display control unit 136 presents either a human face image or a character face image as a face image. It should be noted that the face image presented as a 3D model is merely an image that visualizes the characteristics of the voice, and does not necessarily have to be identical to the voice data of the person whose face image is displayed.

[0084] The following description explains the process for presenting facial images. For example, the display control unit 136 presents a facial image of a person similar to the speech style of each cluster at the center of the coordinates of each cluster, selected from a set of preset facial images of people stored as default in the storage unit 120.

[0085] Alternatively, the display control unit 136 may prepare a model trained to output a face image corresponding to the speech style from the audio data and present the output face image. The display control unit 136 may also train the audio dataset used in the compression and reconstruction training described later to include corresponding face images along with the audio, and present a face image corresponding to the speech style.

[0086] Figure 7 shows an example of displaying face images on a mapping space. As shown in Figure 7, the display control unit 136 presents face images on a mapping space where speech styles divided into clusters are plotted. For example, the display control unit 136 presents face images 71a, 71b, 71c, and 71d, which represent the characteristics of each cluster divided by voice pitch.

[0087] For example, the display control unit 136 displays variations in the intensity of the impression word 31 that specify the speech style for the synthesized speech, in response to the movement of a pointer indicating the position of the impression word 31 in the mapping space.

[0088] For example, the display control unit 136 displays the intensity of the impression word 31 based on the position coordinates indicated by the pointer. The display control unit 136 displays the intensity of the impression word 31 based on the position coordinates of the pointer in response to the movement of the pointer. The position coordinates of the pointer indicating the intensity of the impression word 31 are the element values ​​of a low-dimensional embedding vector.

[0089] The display control unit 136 inputs a low-dimensional embedding vector corresponding to the pointer's position coordinates into a pre-trained reconstructor (described later), and presents speech synthesized using the output high-dimensional speech style embedding vector.

[0090] The display control unit 136 presents the user with synthesized speech generated from the position coordinates at the location moved by the user's pointer movement operation. In other words, the display control unit 136 can change the speech style and present synthesized speech based on the user's pointer operation.

[0091] Figure 8 shows an example of a UI for a user to adjust their speaking style. As shown in Figure 8, the display control unit 136 presents a screen 71 for the user to adjust their speaking style. On the screen 71 for the user to adjust their speaking style, the display control unit 136 presents a screen 72 for receiving input of text that defines the voice to be synthesized.

[0092] Furthermore, the display control unit 136 presents slider bars 73 for adjusting the intensity of the impression words 31. For example, the display control unit 136 presents the user with slider bars corresponding to the impression words 31 of brightness, clarity, and strength, respectively. Note that the slider bars 73 can be added and updated through learning related to subjective evaluation information by the user, as described later. The display control unit 136 may, by default, present slider bars 73 based on any impression word 31.

[0093] The display control unit 136 may present the user with a screen 74 for making additional adjustments, such as fine-tuning the speaking speed and pitch of the spoken text. The display control unit 136 then presents the user with a screen 75 that plays back synthesized speech generated according to the speech style, which includes the impression words 31 adjusted by the slider bar 73, and the text.

[0094] The display control unit 136 presents a screen 76 of the mapping space, which includes a pointer 77 indicating the position of the impression word 31. For example, the display control unit 136 displays the position of the current utterance style in the mapping space on the screen 76 of the mapping space using the pointer 77. The display control unit 136 may also present the slider bar 73 and the pointer 77 in conjunction. Specifically, the display control unit 136 may present the trajectory of the pointer 77's movement in accordance with the movement of the slider bar 73. Alternatively, the display control unit 136 may present instructions to adjust the value of the slider bar 73 in accordance with the movement of the pointer 77.

[0095] The display control unit 136 may present the slider bar 73 and the mapping space screen 76 to a device other than the information processing device. For example, the display control unit 136 may present the mapping space to a device such as a head-mounted display using VR (Virtual Reality) for a full-sphere display or AR (Augmented Reality) for an overhead view display. The display control unit 136 may also present the mapping space on a device such as a head-mounted display so that the user can operate the pointer 77.

[0096] The display control unit 136 may present the generation of adjustment parameters from the speech style embedding vector and the visualization of the mapping space in a hierarchical manner. For example, the display control unit 136 may present the mapping space of speech styles related to "speaker characteristics" as impression words 31, and then present the mapping space of speech styles related to "emotions of a specific speaker" as impression words 31.

[0097] (2-4. Example of the configuration of the learning process according to the embodiment) (2-4-1. Generation of adjustment parameters according to the embodiment) Next, the learning process according to the present disclosure will be described. First, the process of generating adjustment parameters based on subjective evaluation information will be described.

[0098] For example, the reception unit 131 receives impression words 31 from the user as impression words 31 included in subjective evaluation information, which are to be used for voice evaluation.

[0099] For example, the reception unit 131 accepts at least one of the following as impression words 31: speaker type, age, emotional expression, intonation, and dialect. Specifically, the reception unit 131 accepts things like huskyness, pitch, or low voice as speaker type. The reception unit 131 accepts things like a young person's voice or an elderly person's voice as age type. The reception unit 131 accepts things like sad, happy, or joyful as emotional expression type. The reception unit 131 accepts things like rising and falling intonation as intonation type. The reception unit 131 accepts various dialects such as Kansai dialect, Hakata dialect, and Tsugaru dialect.

[0100] For example, the reception unit 131 accepts at least one of the following as impression words 31: the addition of noise and frequency characteristics. For example, the reception unit 131 accepts the addition of noise when the sound quality of the synthesized speech is reduced. For example, the reception unit 131 accepts the removal of high-frequency or low-frequency sounds as frequency characteristics. Specifically, the reception unit 131 accepts the removal of high-frequency or low-frequency sounds when the characteristics of the recording equipment are expressed as a speech style.

[0101] Figure 9 shows an example of receiving impression words 31 to be evaluated on the voice. As shown in Figure 9, the reception unit 131 receives the impression words 31 to be evaluated on the voice from the screen 141 which receives the designation of points (impression words 31) to be evaluated on the voice.

[0102] The screen 141 that accepts the specification of points for evaluating the sound consists of an item 142 for selecting an impression word 31 from a dropdown list and an item 143 for entering an arbitrary impression word 31. Note that the item 142 that accepts the specification of points for evaluating the music may consist of only one of the items 142 for selecting an impression word 31 from a dropdown list and the item 143 for entering an arbitrary impression word 31.

[0103] For example, the reception unit 131 may accept the impression word 31 in item 142, where the user selects the impression word 31 from a dropdown list. Alternatively, the reception unit 131 may accept the impression word 31 in item 143, where the user inputs an arbitrary impression word 31. For example, if the user cannot input an explicit impression word 31, the reception unit 131 may accept a meaningless word such as "parameter1" as the impression word 31 instead of an explicit impression word 31. The storage unit 120 stores multiple impression words 31 in advance as dropdown lists.

[0104] For example, the reception unit 131 receives at least one evaluation from the voice samples of each of the multiple clusters separated by clustering. In other words, the reception unit 131 receives subjective evaluation results (subjective evaluation values) from the voice samples of representative data of the multiple clusters separated by clustering, which will be described later.

[0105] Specifically, the reception unit 131 receives evaluations from the user regarding impression words 31 for the voice (voice sample) corresponding to the centroid vector representing the cluster, which will be described later.

[0106] For example, the reception unit 131 may present the user with multiple audio samples as an evaluation (response value) for the audio data, and accept a score from 0 to 10 for each. Alternatively, the reception unit 131 may present the user with two different audio samples as an evaluation of the audio data, and accept a response indicating which one is better. Note that the evaluation method is not limited to providing a score.

[0107] Figure 10 shows an example of how to receive evaluations of sample audio. As shown in Figure 10, the reception unit 131 receives the score of the sample audio from a screen 144 that receives evaluations of impression words 31 such as "huskyness". The screen 144 that receives evaluations of impression words 31 consists of an item 145 for receiving the score of the sample audio, an item 146 for specifying the playback section of each sample audio, and an item 147 for receiving playback or stopping of each sample audio.

[0108] The reception unit 131 accepts the scores for each of the four sample audio samples in the range of 0 to 10 points using item 146 for accepting scores for each sample audio. The reception unit 131 accepts the playback section for each sample audio sample using item 147 for specifying the playback section for each sample audio sample. The reception unit 131 accepts the playback or stop of each sample audio sample using item 147 for accepting playback or stop. Note that the number of samples is not limited to four.

[0109] For example, the generation unit 132 generates adjustment parameters to adjust the speech style of the low-dimensional embedding vector based on subjective evaluation information. The following description explains how to generate the adjustment parameters.

[0110] The generation unit 132 generates parameters that allow manipulation of impression words 31 from subjective evaluation results (subjective evaluation values). For example, in order to generate adjustment parameters, the generation unit 132 derives a function that predicts subjective evaluation values ​​from the element values ​​of a high-dimensional embedding vector or a low-dimensional embedding vector, and varies each element of the embedding vector in accordance with the increasing direction of the prediction function.

[0111] The generation unit 132 may generate adjustment parameters regardless of the vector dimensions. For example, the generation unit 132 may generate adjustment parameters using high-dimensional embedding vectors, or it may generate adjustment parameters using low-dimensional embedding vectors compressed by the compression unit 135.

[0112] The generation unit 132 generates adjustment parameters that allow manipulation of each element value of a high-dimensional embedding vector or a low-dimensional embedding vector, such that the subjective evaluation value for the impression word 31 increases. For example, the generation unit 132 calculates a function that predicts the subjective evaluation value using the subjective evaluation response results for the sample audio described later.

[0113] The generation unit 132 generates a prediction function for subjective evaluation values ​​using multiple regression analysis. Specifically, in multiple regression analysis, the predictor variable is calculated by determining partial regression coefficients from paired data of multiple observed explanatory variables and the dependent variable.

[0114] For example, in multiple regression analysis, each element of the high-dimensional embedding vector or each element of the low-dimensional embedding vector corresponding to the audio data used to obtain the response values ​​is used as the explanatory variable, and the response value obtained from subjective evaluation is used as the dependent variable. This allows multiple regression analysis to predict subjective evaluation values ​​for any high-dimensional embedding vector or low-dimensional embedding vector having arbitrary element values.

[0115] Figure 11 shows an example of multiple regression analysis. In Figure 11, subjective evaluation scores are plotted as the dependent variable for two explanatory variables. Plane 186 shows the predictive function obtained by multiple regression analysis. Arrow 187 indicates that the subjective evaluation score increases as you move in the direction of arrow 187. Note that Figure 11 is a conceptual diagram illustrating the multiple regression analysis described above.

[0116] In multiple regression analysis, the prediction of the value of the dependent variable for multiple explanatory variables is expressed by the following regression equation (2).

[0117]

[0118] In equation (2) above, y is the predicted value of the dependent variable, p=1...P is the index of the independent variable, xp and βp are the p-th independent variable and partial regression coefficients, respectively, and β0 is the constant term.

[0119] In the above formula, each partial regression coefficient and its sign directly correspond to the direction of increase or decrease in the predicted value. For example, in the above formula, if the sign of βp is positive, increasing xp increases the predicted value, and if the sign is negative, increasing xp decreases the predicted value. In other words, to increase the subjective evaluation value for the specified impression word 31, one should increase each element of the high-dimensional embedding vector or the low-dimensional embedding vector by the absolute value |βp| of each partial regression coefficient.

[0120] The generation unit 132 generates an adjustment parameter that increases the subjective evaluation value 39 for a specified impression word 31 by operating according to the above formula. The generation unit 132 generates an adjustment parameter having a mechanism that increases the value in one dimension, such as a slider bar. The generation unit 132 generates an adjustment parameter that can manipulate the specified impression word 31 by increasing each element value in the above formula by the same unit (for example, 1.0, 1, etc.) in response to the increase in value due to the operation of the slider bar.

[0121] Furthermore, adjustment parameters may be managed and shared on a per-user basis or among users. For example, adjustment parameters may be shared among users for version control and preset purposes. For instance, shared adjustment parameters may be downloaded in order of user count or download count, allowing users to use them in their own environments.

[0122] The clustering unit 134 clusters the high-dimensional embedding vectors extracted from the audio dataset. For example, the clustering unit 134 clusters the high-dimensional embedding vectors obtained from each audio data in the audio dataset, or the low-dimensional embedding vectors obtained by compressing the high-dimensional embedding vectors using a compressor.

[0123] High-dimensional or low-dimensional embedding vectors may be extracted from the audio dataset using existing methods, or they may be extracted using the style encoder described later.

[0124] For example, the clustering unit 134 uses existing techniques such as the K-means method, the K-means++ method, and the EM (expectation maximization) algorithm which assumes a Gaussian mixture model as clustering methods.

[0125] The following explanation uses the K-means method as an example. The K-means method divides data into an arbitrary number of clusters. Then, the K-means method randomly sets the same number of centroid vectors as the number of clusters and updates the centroids of the clusters.

[0126] In the K-means method, an arbitrary number of high-dimensional centroid vectors μk are randomly assigned. k = 1, 2...k are the cluster indices.

[0127] In the K-means method, each vector Xi in the audio dataset is selected to belong to the cluster of centroid vectors that has the shortest distance using the following formula, where i is the index of the vector included in the audio dataset.

[0128]

[0129] In the K-means method, for each of the k clusters obtained using equation (3) above, a new higher-dimensional centroid vector μk is determined from the vectors belonging to the cluster.

[0130] In the K-means method, each vector Xi in the audio dataset is processed again to determine which centroid vector cluster it belongs to. This process is repeated until the vectors belonging to the clusters no longer change.

[0131] The K-means method selects a representative centroid vector (hereinafter also called a representative embedding vector) for a cluster using the method described above.

[0132] Figure 12 shows an example of clustering results. As shown in Figure 12, the clustering unit 134 creates multiple clusters from either high-dimensional or low-dimensional embedding vectors. The clustering unit 134 also selects a cluster centroid vector that represents each cluster.

[0133] For example, the clustering unit 134 clusters the audio dataset based on the frequency characteristics corresponding to the impression words 31. Specifically, the clustering unit 134 may perform clustering according to the impression words 31 specified by the user, as the centroid vector representing the cluster may not possess the characteristics of the impression words 31 specified by the user. For example, the clustering unit 134 clusters the data by dividing it into high-frequency groups and low-frequency groups. The clustering unit 134 performs clustering according to the impression words 31 by associating acoustic features with the impression words 31.

[0134] For example, the clustering unit 134 performs frequency-based clustering by associating impression words 31 related to the pitch of sounds, such as "brightness" and "sparkle," with acoustic features.

[0135] For example, the clustering unit 134 clusters the audio dataset based on the duration of phonemes corresponding to impression words 31. For instance, the clustering unit 134 clusters impression words 31 related to the temporal envelope of speech, such as "slow" or "calm," by associating them with acoustic features, based on the duration of phonemes.

[0136] For example, if subjective evaluation results for a sample audio are obtained from multiple users, the clustering unit 134 may perform clustering based on the subjective evaluation results, by associating the subjective evaluation results of impression words 31 obtained from multiple users with acoustic features. The storage unit 120 stores the response results for the sample audio obtained from multiple users.

[0137] The display control unit 136 presents the user with audio samples synthesized from each of the multiple clusters of the high-dimensional embedding vectors classified by clustering. For example, the display control unit 136 presents the user with audio samples synthesized from centroid vectors (also called representative data) that represent each cluster separated by clustering. The centroid vectors are representative data for each cluster. It is desirable that the representative data for each cluster be selected to summarize the trend of the entire audio dataset and to be distant from each other.

[0138] In the K-means method described above, the centroid vector is calculated independently of the audio dataset, so the audio data corresponding to the representative embedding vector does not exist in the audio dataset. Therefore, the K-means method outputs the audio data corresponding to the embedding vector with the closest Euclidean distance to the representative embedding vector. In addition, the K-means method outputs the synthesized speech obtained by inputting the representative embedding vector into a speech synthesis model as audio data.

[0139] Furthermore, if the display control unit 136 uses a method other than the K-means method, it may present the user with sample audio from the audio dataset.

[0140] (2-4-2. Learning of the speech synthesis model and style encoder according to the embodiment) First, we will explain how to learn the speech synthesis model that synthesizes speech from text and the style encoder that generates embedding vectors from acoustic features such as Mel spectrograms. The learning unit 137 learns the speech synthesis model and the style encoder simultaneously. The learning unit 137 may learn the speech synthesis model and the style encoder separately.

[0141] Figure 13 shows an example of training between the speech synthesis model 301 and the style encoder 302. As shown in Figure 13, the training unit 137 trains the speech synthesis model and the style encoder simultaneously. The speech synthesis model 301 includes g2p 152, a text encoder 153, and a decoder 154.

[0142] The learning unit 137 takes a text 151 that defines the content of the synthesized speech to be output as input and trains g2p 152 to convert it into phonemes. Note that g2p 152 may use a pre-trained model from existing technology.

[0143] The learning unit 137 trains the text encoder 153 to extract features using the converted phonemes as input. Note that the text encoder 153 may use a pre-trained model from existing technology.

[0144] The learning unit 137 trains the decoder 154 to output a Mel spectrogram by taking as input a vector obtained by adding or concatenating the speech style embedding vector to the feature quantity. Note that the decoder 154 may use a pre-trained model from existing technology.

[0145] Incidentally, the style encoder 302 includes a reference encoder 158 and an attention mechanism 159. The learning unit 137 trains the style encoder 302 to output a speech style embedding vector using a spectrogram as input. The training of the style encoder 302 is performed on the speech dataset without explicitly providing a label for the speech style. The spectrogram is extracted as an acoustic feature from the speech waveform 156 included in the speech dataset.

[0146] Specifically, the learning unit 137 takes the Mel spectrogram 157 as input and trains the reference encoder 158 to output a fixed-length vector independent of the length of the audio signal contained in the Mel spectrogram 157. The learning unit 137 also trains the attention mechanism 159 to output a speech style embedding vector 160, taking the fixed-length vector independent of the length of the audio signal as input. The attention mechanism 159 learns the weights that contain important elements of the vector.

[0147] The learning unit 137 uses the generated embedding vector as a condition to learn the model so that the error between the Mel spectrogram 155 generated from the decoder included in the speech synthesis model and the Mel spectrogram 157 used for learning the embedding vector becomes smaller.

[0148] Note that the speech synthesis model and style encoder may be used separately. For example, the speech synthesis model outputs synthesized speech 17 according to a specified speech style using speech style embedding vectors. For example, the style encoder is used to adjust the speech style using low-dimensional embedding vectors rather than high-dimensional embedding vectors.

[0149] (2-4-3. Learning of Compressed Representations According to the Embodiment) Learning of compressed representations refers to compressing a high-dimensional (hereinafter also referred to as n-dimensional) vector into a low-dimensional (hereinafter also referred to as m-dimensional) vector, and then reconstructing a high-dimensional vector from the compressed low-dimensional vector. First, the learning unit 137 extracts high-dimensional vectors from the learning data.

[0150] Figure 14 shows an example of compressed representation learning. As shown in Figure 14, the learning unit 137 extracts acoustic features 167 from any number of audio waveform data 166 obtained from the audio dataset 165. For example, the acoustic features 167 are Mel spectrograms.

[0151] The learning unit 137 extracts acoustic features 167 from waveform data (data 1, data 2, ... data N) 166 according to the number of waveform data (data 1, data 2, ... data N). The learning unit 137 extracts multiple high-dimensional embedding vectors 168 from the features obtained from the waveform data 166. If there are multiple waveform data 166, the learning unit 137 extracts multiple high-dimensional embedding vectors 168 corresponding to each waveform data 166.

[0152] The learning process for extracting high-dimensional embedding vectors from waveform data 166 is performed by the style encoder described above, so its explanation is omitted here.

[0153] The learning unit 137 concatenates multiple high-dimensional embedding vectors 168 to generate an integrated embedding vector (n-dimensional × N matrix). The learning unit 137 uses the integrated embedding vector as training data to train the compressor 171 and the reconstructor. Through training the compressor 171 and the reconstructor, the learning unit 137 obtains compressor model parameters and reconstructor model parameters 170. The compressor model parameters and reconstructor model parameters 170 are used in the compression or reconstruction process in the speech synthesis process (inference process).

[0154] The learning unit 137 uses the compressor model parameters obtained through learning to convert the integrated high-dimensional embedding vector 169 into an integrated low-dimensional embedding vector (m-dimensional × N matrix) 172. Subsequently, since the integrated low-dimensional embedding vector 172 is an embedding vector in which multiple embedding vectors are integrated, the learning unit 137 extracts the low-dimensional embedding vector 173 column by column.

[0155] Next, we will explain in detail the learning process for the compressor 171 and the reconstructor. First, we will explain Truncated Singular Value Decomposition (TSVD) as a dimensionality reduction method.

[0156] Figure 15 shows an example of singular value decomposition. Figure 15 also illustrates equations (4), (5), and (6), which are described below. As shown in Figure 15, equation (4) corresponds to equation 175 in the figure. Equation (5) corresponds to equation 176 in the figure. Equation (6) corresponds to equation 177 in the figure.

[0157] Singular value decomposition performs the following process on a matrix X of size N × n, which is a high-dimensional data set formed by integrating any number of audio waveform data (n). Equations (4), (5), and (6) are mathematical formulas related to the singular value decomposition process.

[0158]

[0159] In equation (4) above, matrices U and V are orthogonal matrices of size N×N and n×n, respectively. Matrix S is a diagonal matrix of size N×n, with the diagonal elements listed in descending order of singular values.

[0160] Each column of matrix V is a singular vector corresponding to a singular value. Since the larger the singular value of a singular value vector, the greater its contribution to the original matrix X, we consider a matrix V(m) of size n × m obtained by truncating matrix V at the first m columns.

[0161] By applying this matrix V(m) to the original matrix X as shown in equation (5), an N×m matrix X' is obtained. In other words, the original N n-dimensional data can be compressed into m dimensions.

[0162]

[0163] Conversely, the m-dimensional compressed matrix X' can be reconstructed into its original n-dimensional form by applying the transpose matrix V(m)T of matrix V(m), as shown in equation (6), to obtain an N×n matrix X^.

[0164]

[0165] In this disclosure, a matrix of n-dimensional embedding vectors can be treated as X, and the matrix V(m) can be considered as a compressor, and its transpose matrix V(m)T as a reconstructor. Furthermore, the element values ​​of matrix V(m) become model parameters.

[0166] The dimensionality reduction method is not limited to compressing high-dimensional vectors by performing linear operations on them using the aforementioned TSVD; other methods that perform linear or nonlinear operations on high-dimensional vectors may also be used.

[0167] For example, other methods that can be selected include Principal Component Analysis (PCA), Kernel Principal Component Analysis, Autoencoder (AE), Variational Autoencoder (VAE), derived autoencoder methods, and Uniform Manifold Approximation and Projection (UMP). These other methods use high-dimensional vectors as training data to learn a compressor that converts to low-dimensional vectors and a reconstructor that converts low-dimensional vectors to high-dimensional vectors.

[0168] Figure 16 shows an example of learning compression and reconstruction. Figure 16 illustrates an example of learning compression and reconstruction using an autoencoder. As shown in Figure 16, vector compression and reconstruction are learned using a compressor (encoder 182) and a reconstructor (decoder 184). Note that encoder 182 and decoder 184 are neural networks.

[0169] The learning unit 137 learns the encoder 182 to compress the high-dimensional vector 181 to be compressed into a low-dimensional vector 183. The learning unit 137 also learns the decoder 184 to reconstruct the low-dimensional vector 183 into a reconstructed high-dimensional vector 185. The learning unit 137 learns the encoder 182 and decoder 184 so that the error between the high-dimensional vector 181 to be compressed and the high-dimensional vector 185 to be reconstructed is minimized.

[0170]

[0171] For example, the learning of the error between the encoder and decoder is expressed by equation (7) above. The high-dimensional vector 181 to be compressed is xi. The high-dimensional vector 185 to be reconstructed is x^i. The learning unit 137 treats the mean squared error (MSE), which is the average value of the squared errors between xi and x^i across the entire learning data, as the loss function L, and learns the model parameters of the encoder 182 and decoder 184 to minimize this loss function. Here, N is the total number of learning data, and i=1, ..., N are the data indices.

[0172] Next, the number of dimensions after compression will be explained. If the number of dimensions after compression is set to the default or to a setting arbitrarily made by the user, the learning unit 137 may learn to compress to the specified number of dimensions.

[0173] The learning unit 137 learns the compressor according to the method of manipulating the low-dimensional vector after compression using adjustment parameters. For example, the learning unit 137 learns the compressor according to the case where multiple elements are manipulated simultaneously in a predetermined combination and the case where each of the multiple elements is manipulated directly. The predetermined combination refers to a combination of multiple groups obtained by further dividing the multiple elements into one group or multiple groups.

[0174] When the learning unit 137 directly manipulates each of the multiple elements, it is desirable that it learns to compress the speech style to as few dimensions as possible (for example, 10 dimensions or less) because it adjusts the speech style by directly manipulating each of the multiple elements. In other words, the learning unit 137 learns to compress to a lower dimension in order to reduce the burden on the user to adjust the speech style by adjusting with a small number of dimensions.

[0175] On the other hand, when the learning unit 137 manipulates multiple elements simultaneously, it does not need to perform training to reduce the number of dimensions after compression to a small number such as 10 dimensions or less, because it moves multiple elements simultaneously using a single-axis parameter during speech style adjustment.

[0176] Incidentally, the learning unit 137 may learn to compress speech style embedding vectors according to various standards for transmitting parameters. For example, with MPEG (Moving Picture Experts Group)-I Immersive Audio, a use case is envisioned for communication between avatars in a virtual space in "Social VR".

[0177] In communication between avatars in a virtual space, the synthesized speech 17, which specifies the speech style as described in this disclosure, is used to allow each avatar to speak in a variety of speech styles. Here, the parameters that determine the speech style (e.g., speech style embedding vectors) are transmitted to an external device. An example of the syntax for transmitting the speech style is described below.

[0178] For example, the learning unit 137 prepares multiple compression patterns for high-dimensional speech style embedding vectors. Then, the learning unit 137 handles a flag specifying the compression pattern and the values ​​of each parameter as transmission information. Figure 17 shows an example of transmission information. As shown in Figure 17, the learning unit 137 generates a flag specifying the compression pattern, "compression pattern, 3 bits," and the transmission capacity of each parameter, "parameter 1, 64 bits... parameter 5, 64 bits," as transmission information 201.

[0179] The above describes an example configuration of the information processing device 100, but the information processing system in this disclosure may consist of a terminal device 10 or the like, in addition to the information processing device 100. The reception unit 131 and the display control unit 136 described above may be configured in other devices. For example, the information processing system includes the information processing device 100 and the terminal device 10, and at least one of the information processing device 100 and the terminal device 10 includes the reception unit 131 and the display control unit 136.

[0180] The configuration of the information processing system is merely an example; the terminal device 10 may include a generation unit 132, or it may be composed of other devices besides the terminal device 10.

[0181] (2-5. Example of the Configuration of an Information Processing System According to an Embodiment) Figure 18 shows an example of the configuration of an information processing system according to an embodiment. The information processing system according to an embodiment consists of an information processing device 100 and a terminal device 10. The following description will explain an example of processing performed by the information processing device 100 and the terminal device 10. Note that redundant explanations will be omitted.

[0182] The terminal device 10 is an information processing device used by the user. The terminal device 10 may be, for example, a smartphone, a tablet device, a desktop PC (Personal Computer), a notebook PC, a mobile phone, a PDA (Personal Digital Assistant), an HMD (Head Mounted Display), a game device, or other information processing device.

[0183] As shown in Figure 18, the terminal device 10 receives user operation results 191. For example, the terminal device 10 receives text information defining the speech to be synthesized (spoken voice) as the user operation results 191. The terminal device 10 also receives impression words 31 as subjective evaluation information and adjustments to the impression words 31 using adjustment parameters.

[0184] The terminal device 10 presents the audio waveform 192 to the user. Alternatively, the terminal device 10 may present synthesized speech to the user instead of the audio waveform 192.

[0185] The information processing device 100 performs processing other than that performed by the terminal device 10 described above in the embodiment, such as training a model 195, including a speech synthesis model and a style encoder, and speech synthesis 194, according to the specified speech style.

[0186] Incidentally, the terminal device 10 may perform some of the processing that the information processing device 100 shown in Figure 18 performs. Figure 19 shows an example of the configuration of an information processing system according to an embodiment.

[0187] As shown in Figure 19, the terminal device 10 may perform a process to reconstruct a speech style embedding vector from a low-dimensional embedding vector using the parameters of a trained reconstructor stored in the information processing device 100 (speech style embedding vector reconstruction 193 in the figure). Alternatively, the terminal device 10 may perform a process to synthesize speech using the parameters of a trained speech synthesis model stored in the information processing device 100 (speech synthesis 194 in the figure).

[0188] The information processing device 100 performs processing other than that performed by the terminal device 10 described above in this disclosure, such as training models 195, including speech synthesis models and style encoders.

[0189] Furthermore, the processing according to this embodiment may be performed by either the information processing device 100 or the terminal device 10 alone. Also, the processing performed by the information processing device 100 and the terminal device 10 is not limited to the examples described above.

[0190] (3. Information Processing Procedure According to the Embodiment) Next, the processing flow of the information processing device 100 will be described using Figures 20, 21, and 22. Figure 20 is a flowchart showing an example of a processing procedure for generating synthesized speech according to the embodiment.

[0191] As shown in Figure 20, the receiving unit 131 of the information processing device 100 determines whether it has received text information from the user that defines the speech to be synthesized (speech content) (step S101). The receiving unit 131 repeats the process in step S101 until it receives text information (step S101: No).

[0192] If the speech synthesis unit 133 acquires text information (step S101: Yes), it extracts features from the text information (step S102). Subsequently, the speech synthesis unit 133 acquires a speech style embedding vector (step S103). The speech synthesis unit 133 repeats the process in step S103 until it acquires a speech style embedding vector (step S103: No).

[0193] Here, we will describe the procedure for generating speech style embedding vectors in order to accept them. Figure 21 is a flowchart showing an example of the procedure for generating speech style embedding vectors according to an embodiment.

[0194] As shown in Figure 21, the reception unit 131 of the information processing device 100 determines whether it has received a specification of speech style from the user (step S201). For example, the reception unit 131 receives a specification of impression word 31 as a speech style. The reception unit 131 repeats the process of step S201 until it receives a specification of speech style from the user (step S201: No).

[0195] If the reception unit 131 receives a specification of speech style from the user (step S201: Yes), it determines whether it has received a request from the user to adjust the speech style using adjustment parameters (step S202). The reception unit 131 repeats the process in step S202 until it receives a request from the user to adjust the speech style using adjustment parameters (step S202: No).

[0196] When the generation unit 132 receives a request from the user to adjust the speech style using adjustment parameters (step S202: Yes), it generates a low-dimensional embedding vector based on the specified speech style and the adjustments made using the adjustment parameters (step S203).

[0197] Next, the generation unit 132 restores the generated low-dimensional embedding vector into a high-dimensional speech style embedding vector (step S204).

[0198] Returning to the explanation of Figure 20, if the speech synthesis unit 133 obtains a speech style embedding vector (step S103: Yes), it determines whether the speech style embedding vector has been added to or concatenated with a vector obtained by extracting features from text information (step S104).

[0199] The speech synthesis unit 133 repeats the process in step S104 until the speech style embedding vector is added to or concatenated with the vector obtained by extracting features from the text information (step S104: No).

[0200] The speech synthesis unit 133 generates synthesized speech (step S105) when speech style embedding vectors are added or concatenated (step S104: Yes).

[0201] Next, the procedure for generating adjustment parameters will be described. Figure 22 is a flowchart showing an example of the procedure for generating adjustment parameters according to the embodiment.

[0202] As shown in Figure 22, the receiving unit 131 of the information processing device 100 determines whether it has received a designation of impression word 31 from the user (step S301). The receiving unit 131 repeats the process in step S301 until it receives a designation of impression word 31 from the user (step S301: No).

[0203] When the generation unit 132 receives a specification of impression words 31 from the user (step S301: Yes), it extracts a high-dimensional vector representing the speech features obtained from the speech dataset (step S302).

[0204] Next, the clustering unit 134 clusters the high-dimensional vectors (step S303). The clustering unit 134 selects representative data from the clustered high-dimensional vectors (step S304).

[0205] The display control unit 136 presents the user with a representative audio sample from the selected audio (step S305). The reception unit 131 determines whether it has received subjective evaluation information regarding the evaluation of the audio sample from the user (step S306). The reception unit 131 repeats the process in step S306 until it receives subjective evaluation information from the user (step S306: No).

[0206] When the reception unit 131 receives subjective evaluation information from a user (step S306: Yes), it determines whether it has obtained subjective evaluation information from the user a predetermined number of times (step S307). The reception unit 131 repeats the process from step S305 until it has obtained subjective evaluation information from the user a predetermined number of times (step S307: No). The predetermined number of times can be set by the user at any value, or it may be set by default to an arbitrary value such as 5 or 10 times.

[0207] When the generation unit 132 obtains subjective evaluation information from the user a predetermined number of times (step S307: Yes), it generates adjustment parameters from the calculated subjective evaluation values ​​(step S308).

[0208] (4. Other Embodiments) The processes according to each of the embodiments described above may be carried out in various other forms besides those described above.

[0209] Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0210] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0211] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent.

[0212] Furthermore, the effects described herein are merely illustrative and not limiting; other effects may also occur.

[0213] (5. Effects of the Information Processing Device According to the Disclosure) As described above, the information processing device according to the Disclosure (information processing device 100 in the embodiment) comprises a receiving unit (receiving unit 131 in the embodiment), a generation unit (132 in the embodiment), a speech synthesis unit (133 in the embodiment), a clustering unit (clustering unit 134 in the embodiment), and a display control unit (136 in the embodiment). The receiving unit receives subjective evaluation information regarding the evaluation of speech and a specification of the speech style from the user. Based on the subjective evaluation information and the specification of the speech style, the generation unit converts a low-dimensional embedding vector corresponding to the speech style adjusted by the user to generate a high-dimensional speech style embedding vector. The speech synthesis unit synthesizes speech from the speech style embedding vector and text information defining the speech to be synthesized.

[0214] Thus, the information processing device relating to this disclosure allows for the adjustment of speech style in a short amount of time by specifying and adjusting speech style at a low level. Furthermore, because the information processing device adjusts speech style based on subjective evaluation information corresponding to the user's subjective feelings and impressions, it can obtain adjustment results that are satisfactory to the user.

[0215] Furthermore, the information processing device relating to this disclosure can adjust the speech style to suit the user in a short amount of time, and can therefore provide synthesized speech that matches the speaker's speech style. In other words, the information processing device can improve the usability of adjusting the speech style.

[0216] Furthermore, the generation unit generates adjustment parameters to adjust the speech style of the low-dimensional embedding vectors based on subjective evaluation information.

[0217] Thus, the information processing device relating to this disclosure allows for adjustment of speech style at a low level, making it easy to adjust speech style.

[0218] The reception desk also accepts the specification of one or more impression words that define the speaking style, and the adjustment of the specified impression words using adjustment parameters.

[0219] Thus, the information processing device relating to this disclosure can present synthesized speech to the user that corresponds to the speech style by adjusting the speech style.

[0220] The clustering unit clusters the speech style embedding vectors into multiple clusters.

[0221] Thus, the information processing device relating to this disclosure can visualize speech styles by dividing speech style embedding vectors into multiple clusters.

[0222] The display control unit plots the numerical vector variables that constitute the speech style embedding vectors separated by clustering onto the mapping space in the UI.

[0223] Thus, the information processing device relating to this disclosure can visually present the distribution of speech styles.

[0224] Furthermore, the display control unit presents a face image or a 3D model in the UI for at least one of the variables plotted on the mapping space.

[0225] Thus, the information processing device relating to this disclosure can visually present the speech style to the user by displaying facial images or 3D models corresponding to the characteristics of the speech on a mapping space.

[0226] Furthermore, the display control unit displays variations in the intensity of impression words that specify the speech style for the synthesized speech, in response to the movement of a pointer indicating the position of the impression word in the mapping space.

[0227] Thus, the information processing device relating to this disclosure allows the user to visually manipulate the speech style. Therefore, the information processing device makes it possible to adjust the speech style visually, intuitively, and easily.

[0228] Furthermore, the reception desk accepts impression words from users as words to be used in the evaluation of voice, which are included in the subjective evaluation information.

[0229] Thus, the information processing device relating to this disclosure can adjust the speech style according to the user's subjective opinion by receiving impression words as subjective evaluation information from the user and performing a subjective evaluation of those impression words before adjusting the speech style.

[0230] Furthermore, the reception desk accepts at least one of the following as impressionistic words: speaker, age, emotional expression, intonation, and dialect.

[0231] Thus, the information processing device relating to this disclosure can accept emotional expressions and other such expressions, which may be evaluated differently by each user, as impression words, and can adjust the speech style according to the user's subjectivity. Furthermore, the information processing device can present the user with synthesized speech that corresponds to the said speech style.

[0232] Furthermore, the reception desk accepts at least one of the following as impression words: the addition of noise, and frequency characteristics.

[0233] Thus, the information processing device relating to this disclosure can adjust the speech style to suit the surrounding environment and recording equipment such as a microphone, using impression words as a basis. Furthermore, the information processing device can present the user with synthesized speech corresponding to that speech style.

[0234] Furthermore, the clustering unit clusters the high-dimensional embedding vectors extracted from the audio dataset.

[0235] Thus, the information processing device relating to this disclosure can present to the user voice samples selected from each clustered cluster, as described later, by clustering a voice dataset.

[0236] The display control unit presents the user with audio samples synthesized from each of the multiple clusters of high-dimensional embedding vectors separated by clustering.

[0237] Thus, the information processing device relating to this disclosure can accurately perform a subjective evaluation of speech style by presenting the user with multiple voice samples having different characteristics.

[0238] Furthermore, the clustering unit clusters the audio dataset based on the frequency characteristics corresponding to the impression words.

[0239] Thus, the information processing device relating to this disclosure can perform subjective evaluations of speech styles that improve user convenience by narrowing down clustering according to impression words. The information processing device can facilitate subjective evaluations by users through clustering according to the pitch of the voice.

[0240] Furthermore, the clustering unit clusters the audio dataset based on the duration of phonemes corresponding to impression words.

[0241] Thus, the information processing device relating to this disclosure can perform subjective evaluations of speech styles that improve user convenience by narrowing down clustering according to impression words. The information processing device can facilitate subjective evaluations by users through clustering according to factors such as the speech rate of the audio relative to the text.

[0242] Furthermore, the reception desk receives at least one evaluation from each of the voice samples in the multiple clusters that have been separated by clustering.

[0243] Thus, the information processing device relating to this disclosure can adjust the speech style in accordance with the user's subjective evaluation by receiving evaluations for each cluster.

[0244] The information processing system includes an information processing device and a terminal device, and at least one of the information processing device and the terminal device comprises a reception unit and a display control unit.

[0245] Thus, the information processing device 100 according to this disclosure can receive speech style adjustments from a terminal device 10 or the like and present synthesized speech, thereby improving the usability of speech style adjustments.

[0246] (6. Hardware Configuration) The information devices such as the information processing device 100 and terminal device 10 according to each embodiment described above are realized by a computer 1000 having a configuration such as that shown in Figure 23. The following explanation will use the information processing device 100 according to the embodiment as an example. Figure 23 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the information processing device 100. The computer 1000 has a CPU 1100, RAM 1200, ROM (Read Only Memory) 1300, HDD (Hard Disk Drive) 1400, communication interface 1500, and input / output interface 1600. The various parts of the computer 1000 are connected by a bus 1050.

[0247] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400 and controls each part. For example, the CPU 1100 loads the programs stored in the ROM 1300 or HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0248] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) that are executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0249] The HDD 1400 is a computer-readable recording medium that non-temporarily stores programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that stores a conversion program according to this disclosure, which is an example of program data 1450.

[0250] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (e.g., the Internet). For example, the CPU 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0251] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard or mouse via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as a display, speaker, or printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

[0252] For example, when the computer 1000 functions as an information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes functions such as the control unit 130 by executing an information processing program loaded on the RAM 1200. The HDD 1400 stores the information processing program according to this disclosure and data in the storage unit 120. The CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, these programs may be obtained from other devices via an external network 1550.

[0253] Furthermore, this technology can also take the following configurations: (1) An information processing system comprising: a receiving unit that receives subjective evaluation information regarding the evaluation of speech and a specification of speech style from a user; a generating unit that converts a low-dimensional embedding vector corresponding to the speech style adjusted by the user based on the subjective evaluation information and the specification of speech style to generate a high-dimensional speech style embedding vector; and a speech synthesis unit that synthesizes speech from the speech style embedding vector and text information defining the speech to be synthesized. (2) The information processing system according to (1), wherein the generating unit generates adjustment parameters for adjusting the speech style of the low-dimensional embedding vector based on the subjective evaluation information. (3) The information processing system according to (2), wherein the receiving unit receives the specification of one or more impression words that specify the speech style and the adjustment of the specified impression words by the adjustment parameters. (4) The information processing system according to any one of (1) to (3), further comprising a clustering unit that clusters the speech style embedding vector into a plurality of clusters. (5) The information processing system according to (4), further comprising a display control unit that plots and presents the variables of the numerical vectors constituting the speech style embedding vectors separated by the clustering on a mapping space in the UI. (6) The information processing system according to (5), wherein the display control unit presents a face image or a 3D model on at least one of the variables plotted on the mapping space in the UI. (7) The information processing system according to (5), wherein the display control unit presents a variation in the intensity of the impression word that specifies the speech style for the voice to be synthesized in accordance with the movement of a pointer indicating the position of the impression word on the mapping space. (8) The information processing system according to any one of (1) to (7), wherein the receiving unit receives impression words from the user that are to be evaluated as impression words included in the subjective evaluation information. (9) The information processing system according to (3) or (8), wherein the receiving unit receives at least one of the following as impression words: speaker, age, emotional expression, intonation, and dialect.(10) The information processing system according to (3) or (8), wherein the receiving unit receives at least one of the following as impression words: the addition of noise and frequency characteristics. (11) The information processing system according to any one of (1) to (10), further comprising a clustering unit for clustering high-dimensional embedding vectors extracted from a speech dataset. (12) The information processing system according to (11), further comprising a display control unit for presenting to the user speech samples synthesized from each of a plurality of clusters of high-dimensional embedding vectors separated by the clustering. (13) The information processing system according to (11), wherein the clustering unit clusters the speech dataset based on frequency characteristics corresponding to impression words. (14) The information processing system according to (11), wherein the clustering unit clusters the speech dataset based on the duration of phonemes corresponding to impression words. (15) The information processing system according to (11), wherein the receiving unit receives at least one evaluation from voice samples of each cluster among a plurality of clusters separated by the clustering. (16) The information processing system according to (5) or (12), wherein the information processing system includes an information processing device and a terminal device, and at least one of the information processing device and the terminal device comprises the receiving unit and the display control unit. (17) An information processing method comprising a computer receiving subjective evaluation information relating to the evaluation of voice and a specification of speech style from a user, converting a low-dimensional embedding vector corresponding to a speech style adjusted by the user based on the subjective evaluation information and the specification of speech style to generate a high-dimensional speech style embedding vector, and synthesizing voice from the speech style embedding vector and text information defining the voice to be synthesized.(18) An information processing program for causing a computer to function as an information processing device comprising: a receiving unit that receives subjective evaluation information relating to the evaluation of speech and a specification of speech style from a user; a generating unit that generates a high-dimensional speech style embedding vector by converting a low-dimensional embedding vector corresponding to a speech style adjusted by the user based on the subjective evaluation information and the specification of speech style; and a speech synthesis unit that synthesizes speech from the speech style embedding vector and text information defining the speech to be synthesized.

[0254] 10 Terminal device 100 Information processing device 110 Communication unit 120 Storage unit 121 Model storage unit 130 Control unit 131 Reception unit 132 Generation unit 133 Speech synthesis unit 134 Clustering unit 135 Compression unit 136 Display control unit 137 Learning unit

Claims

1. An information processing system comprising: a receiving unit that receives subjective evaluation information regarding the evaluation of speech and a specification of speech style from a user; a generating unit that converts a low-dimensional embedding vector corresponding to the speech style adjusted by the user based on the subjective evaluation information and the specification of speech style to generate a high-dimensional speech style embedding vector; and a speech synthesis unit that synthesizes speech from the speech style embedding vector and text information defining the speech to be synthesized.

2. The information processing system according to claim 1, wherein the generation unit generates adjustment parameters for adjusting the speech style of the low-dimensional embedding vector based on the subjective evaluation information.

3. The information processing system according to claim 2, wherein the receiving unit receives the specification of one or more impression words that specify the speech style, and the adjustment of the specified impression words using the adjustment parameters.

4. The information processing system according to claim 1, further comprising a clustering unit for clustering the speech style embedding vectors into a plurality of clusters.

5. The information processing system according to claim 4, further comprising a display control unit that plots and presents the variables of the numerical vectors constituting the speech style embedding vectors separated by the clustering on a mapping space in the UI.

6. The information processing system according to claim 5, wherein the display control unit presents a face image or a 3D model in the UI for at least one of the variables plotted on the mapping space.

7. The information processing system according to claim 6, wherein the display control unit presents variations in the intensity of impression words that specify the speech style for the speech to be synthesized, in accordance with the movement of a pointer indicating the position of impression words in the mapping space.

8. The information processing system according to claim 1, wherein the receiving unit receives impression words from the user as impression words to be evaluated by voice, as impression words included in the subjective evaluation information.

9. The information processing system according to claim 8, wherein the receiving unit receives at least one of the following as impression words: speaker, age, emotional expression, intonation, and dialect.

10. The information processing system according to claim 8, wherein the receiving unit receives at least one of the following as impression words: the addition of noise and frequency characteristics.

11. The information processing system according to claim 1, further comprising a clustering unit for clustering high-dimensional embedding vectors extracted from an audio dataset.

12. The information processing system according to claim 11, further comprising a display control unit that presents to the user voice samples synthesized from each of the multiple clusters of high-dimensional embedding vectors separated by the clustering.

13. The information processing system according to claim 11, wherein the clustering unit clusters the audio dataset based on frequency characteristics corresponding to impression words.

14. The information processing system according to claim 11, wherein the clustering unit clusters the audio dataset based on the duration of phonemes corresponding to impression words.

15. The information processing system according to claim 11, wherein the receiving unit receives at least one evaluation from the voice samples of each cluster among the multiple clusters separated by the clustering.

16. The information processing system according to claim 6, wherein the information processing system includes an information processing device and a terminal device, and at least one of the information processing device and the terminal device comprises a receiving unit and a display control unit.

17. An information processing method comprising: a computer receiving subjective evaluation information regarding the evaluation of speech and a specification of speech style from a user; converting a low-dimensional embedding vector corresponding to the speech style adjusted by the user based on the subjective evaluation information and the specification of speech style to generate a high-dimensional speech style embedding vector; and synthesizing speech from the speech style embedding vector and text information defining the speech to be synthesized.

18. An information processing program for causing a computer to function as an information processing device comprising: a receiving unit that receives subjective evaluation information regarding the evaluation of speech and a specification of speech style from a user; a generating unit that converts a low-dimensional embedding vector corresponding to a speech style adjusted by the user based on the subjective evaluation information and the specification of speech style to generate a high-dimensional speech style embedding vector; and a speech synthesis unit that synthesizes speech from the speech style embedding vector and text information defining the speech to be synthesized.

Images