Conversion learning device, conversion learning method, conversion learning program, and conversion device

Abstract

This conversion learning device comprises: a source encoding unit that uses a first machine learning model to convert a source domain feature quantity series, which is a feature of conversion source content data, into a first internal representation vector series, which is a matrix in which internal representation vectors for each location of the source domain feature quantity series are arranged; a target encoding unit that uses a second machine learning model to convert a target domain feature quantity series, which is a feature of conversion target content data, into a second internal representation vector series, which is a matrix in which internal representation vectors for each location of the target domain feature quantity series are arranged; an attention matrix calculation unit that uses the first internal representation vector series and the second internal representation vector series to calculate an attention matrix, which is a matrix in which each of the locations of the source domain feature quantity series and each of the locations of the target domain feature quantity series are associated, and calculates a third internal representation vector series, which is the product of the attention matrix and an internal representation vector series calculated by performing a linear conversion of the first internal representation vector series; a target decoding unit that uses the third internal representation vector series to calculate a conversion domain feature quantity series with a third machine learning model, said conversion domain feature quantity series being used for converting the source domain to a conversion domain; and a learning implementation unit that causes at least one of the target encoding unit and the target decoding unit to perform learning such that the distance between a partial matrix of the target domain feature quantity series and a partial matrix of the conversion domain feature quantity series becomes shorter.

Description

[Technical field]

[0001] The present invention relates to a conversion learning device, a conversion learning method, a conversion learning program, and a conversion device. [Background technology]

[0002] A technology called voice conversion, which converts non-linguistic and paralinguistic information into other values ​​while preserving the linguistic information contained in speech (the character string of a spoken sentence), has been attracting attention. By performing voice conversion, for example, it is possible to generate a voice in which the speaker's voice quality (one of the speaker characteristics) is changed while preserving the linguistic information as it is. The paralanguage that can be converted by voice conversion is diverse, including speaker characteristics, speaking style, accent, and voice quality. For this reason, voice conversion can be applied to various fields such as speech support and voice recognition.

[0003] Voice conversion can be formulated as a regression analysis problem that estimates a mapping function from the features of the source voice to the features of the target voice. For example, a method using a Gaussian Mixture Model (GMM) is highly effective and versatile and is widely used. In recent years, methods using a neural network (NN) and a non-negative matrix factorization (NMF) have also been studied.

[0004] However, the above-mentioned methods are all aimed at converting voice quality, and are not necessarily suitable for converting prosodic features. The prosodic features referred to here are sometimes called suprasegmental features, and are important elements that characterize a speaker's personality and speaking style, such as fundamental frequency patterns, speech rate, and speech rhythm. For this reason, there is a demand for the development of voice conversion that can effectively convert not only voice quality but also prosodic features.

[0005] Furthermore, as disclosed in Non-Patent Documents 1, 2, and 3, development of speech conversion using a sequence-to-sequence (S2S) learning model is underway, and it has been shown to be highly effective in tasks such as machine translation, speech recognition, and text-to-speech synthesis. The sequence-to-sequence learning model is a type of deep learning that learns a conversion function from a sequence to a sequence. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] H. Miyoshi, Y. Saito, S. Takamichi, and H. Saruwatari, “Voice conversion using sequence-to-sequence learning of context posterior probabilities,” in Proc. Interspeech, 2017, pp. 1268-1272. [Non-Patent Document 2] J.-X. Zhang, Z.-H. Ling, L.-J. Liu, Y. Jiang, L.-R. Dai, “Sequence-to-sequence acoustic modeling for voice conversion,”arXiv:1810.06865 [cs.SD], Oct. 2018. [Non-Patent Document 3] F. Biadsy, RJ Weiss, PJ Moreno, D. Kanvesky, and Y. Jia, “Parrotron: An end-to-end speech-to-speech conversion model and its applications to hearing-impaired speech and speech separation,” in Proc. Interspeech, 2019, pp. 4115-4119.

[0007] The principle of voice conversion can also be applied to image conversion and video conversion, for example. For this reason, there is a demand for development of technology that can effectively perform image conversion and video conversion by applying the above-mentioned techniques related to voice conversion. Summary of the Invention [Problem to be solved by the invention]

[0008] The present invention has been made in consideration of the above circumstances, and aims to provide a conversion learning device, a conversion learning method, a conversion learning program, and a conversion device that can perform effective conversion processing. [Means for solving the problem]

[0009] According to one aspect of the present invention, there is provided a source coding unit that converts a source domain feature sequence, which is a feature of content data to be converted, into a first internal representation vector sequence that is a matrix in which internal representation vectors at each position of the source domain feature sequence are arranged, using a first machine learning model; a target coding unit that converts a target domain feature sequence, which is a feature of content data to be converted, into a second internal representation vector sequence that is a matrix in which internal representation vectors at each position of the target domain feature sequence are arranged, using a second machine learning model; and a coding unit that converts each position of the source domain feature sequence into a second internal representation vector sequence that is a matrix in which internal representation vectors at each position of the target domain feature sequence, using the first internal representation vector sequence and the second internal representation vector sequence. a target decoding unit that uses the third internal representation vector sequence to calculate a feature sequence of the transformation domain, which is used to transform the source domain into a transformation domain, using the third internal representation vector sequence, by a third machine learning model; and a learning execution unit that trains at least one of the target encoding unit and the target decoding unit so that a distance between a submatrix of the feature sequence of the target domain and a submatrix of the feature sequence of the transformation domain is shortened.

[0010] According to one aspect of the present invention, there is provided a source coding unit that converts a feature amount sequence of a first domain, which is a feature of source content data and is assigned a first index, into a first internal representation vector sequence that is a matrix in which internal representation vectors at each position of the feature amount sequence of the first domain are arranged, using a first machine learning model; a target coding unit that converts a feature amount sequence of a second domain, which is a feature of target content data and is assigned a second index different from the first index, into a second internal representation vector sequence that is a matrix in which internal representation vectors at each position of the feature amount sequence of the second domain are arranged, using a second machine learning model; and a target decoding unit that calculates a feature sequence of the transformation domain, which is used to transform the first domain into a transformation domain using the third internal representation vector sequence, by a third machine learning model; and a learning execution unit that trains at least one of the target encoding unit and the target decoding unit so that a distance between a submatrix of the feature sequence of the second domain and a submatrix of the feature sequence of the transformation domain is shortened.

[0011] One aspect of the present invention includes a source encoding step of converting a source domain feature sequence, which is a conversion source, into a first internal representation vector sequence, which is a matrix in which internal representation vectors at each position of the source domain feature sequence are arranged, using a first machine learning model; a target encoding step of converting a target domain feature sequence, which is a conversion target, into a second internal representation vector sequence, which is a matrix in which internal representation vectors at each position of the target domain feature sequence are arranged, using a second machine learning model; and calculating an attention matrix, which is a matrix that associates each position of the source domain feature sequence with each position of the target domain feature sequence, using the first internal representation vector sequence and the second internal representation vector sequence. a target decoding step of calculating a feature sequence of the transformation domain, which is used to transform the source domain into a transformation domain using the third internal representation vector sequence, by a third machine learning model; and a learning execution step of training at least one of a target encoding unit having the first machine learning model and a target decoding unit having the second machine learning model so that a distance between a submatrix of the feature sequence of the target domain and a submatrix of the feature sequence of the transformation domain is shortened.

[0012] One aspect of the present invention includes a source encoding step of converting a feature amount sequence of a first domain, which is a feature of source content data and is assigned a first index, into a first internal representation vector sequence that is a matrix in which internal representation vectors at each position of the feature amount sequence of the first domain are arranged, using a first machine learning model; a target encoding step of converting a feature amount sequence of a second domain, which is a feature of target content data and is assigned a second index different from the first index, into a second internal representation vector sequence that is a matrix in which internal representation vectors at each position of the feature amount sequence of the second domain are arranged, using a second machine learning model; and a target decoding step of calculating, using a third machine learning model, a feature sequence of the transformation domain to be used for transforming the first domain into a transformation domain using the third internal representation vector sequence; and a learning execution step of training at least one of a target encoding unit having the first machine learning model and a target decoding unit having the second machine learning model so that a distance between a submatrix of the feature sequence of the second domain and a submatrix of the feature sequence of the transformation domain is shortened.

[0013] One aspect of the present invention is a conversion device that includes a source encoding unit that converts a source domain feature sequence, which is a feature of content data to be converted, into a first internal representation vector sequence that is a matrix in which internal representation vectors at each position of the source domain feature sequence are arranged using a first machine learning model; a target encoding unit that converts a target domain feature sequence, which is a feature of content data to be converted, into a second internal representation vector sequence that is a matrix in which internal representation vectors at each position of the target domain feature sequence are arranged using a second machine learning model; an attention matrix calculation unit that calculates an attention matrix, which is a matrix that associates each position of the source domain feature sequence with each position of the target domain feature sequence, using the first internal representation vector sequence and the second internal representation vector sequence, and calculates a third internal representation vector sequence that is the product of the attention matrix and the internal representation vector sequence calculated by linearly transforming the first internal representation vector sequence; and a target decoding unit that calculates a feature sequence of the conversion domain used to convert the source domain to a conversion domain using the third internal representation vector sequence using the third machine learning model.

[0014] One aspect of the present invention is a conversion device comprising: a source coding unit that converts a feature sequence of a first domain, which is a feature of source content data and is assigned a first index, into a first internal representation vector sequence that is a matrix in which internal representation vectors at each position of the feature sequence of the first domain are arranged using a first machine learning model; a target coding unit that converts a feature sequence of a second domain, which is a feature of target content data and is assigned a second index different from the first index, into a second internal representation vector sequence that is a matrix in which internal representation vectors at each position of the feature sequence of the second domain are arranged using a second machine learning model; an attention matrix calculation unit that calculates an attention matrix, which is a matrix that associates each position of the feature sequence of the first domain with each position of the feature sequence of the second domain, using the first internal representation vector sequence and the second internal representation vector sequence, and calculates a third internal representation vector sequence that is the product of the attention matrix and the internal representation vector sequence calculated by linearly transforming the first internal representation vector sequence; and a target decoding unit that calculates a feature sequence of the conversion domain used to convert the first domain into a conversion domain using the third internal representation vector sequence using the third machine learning model. Effect of the Invention

[0015] According to the present invention, an effective conversion process can be performed. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating an example of a transformation learning device according to a first embodiment. [Diagram 2] 1A to 1C are diagrams for explaining repetition of processes executed by a target encoding unit, an attention matrix calculation unit, and a target decoding unit according to the first embodiment. [Diagram 3] 10 is a flowchart illustrating an example of processing executed by the transformation learning device according to the first embodiment in a learning phase. [Figure 4]10 is a flowchart illustrating an example of a process executed by the conversion device according to the first embodiment in an inference phase. [Diagram 5] A figure showing an example of an attention matrix when the conversion device according to the first embodiment does not perform forward attention correction in the inference phase. [Figure 6] A figure showing an example of an attention matrix when the conversion device according to the first embodiment performs forward attention correction in the inference phase. [Figure 7] 11A to 11C are diagrams illustrating an example of mel cepstral coefficients when the conversion device according to the first embodiment does not perform forward attention correction and an example of mel cepstral coefficients when the conversion device performs forward attention correction. [Figure 8] FIG. 13 is a diagram illustrating an example of a transformation learning device according to a second embodiment. [Figure 9] 13 is a flowchart illustrating an example of processing executed by the transformation learning device according to the second embodiment in a learning phase. [Figure 10] 10 is a flowchart illustrating an example of a process executed by the conversion device according to the second embodiment in an inference phase. [Figure 11] FIG. 11 is a diagram showing an example of an attention matrix in the inference phase when the source encoding unit, the target encoding unit, the target decoding unit, and the target reconstruction unit according to the second embodiment have normal batch normalization layers. [Figure 12] FIG. 11 is a diagram showing an example of an attention matrix in the inference phase when the source coding unit, the target coding unit, the target decoding unit, and the target reconstruction unit according to the second embodiment have batch normalization layers corresponding to the first domain or the second domain. [Figure 13] 13A to 13C are diagrams illustrating an example of mel-cepstral coefficients when the conversion device according to the second embodiment performs one-to-one conversion and many-to-one conversion in the inference phase. [Figure 14] 13A to 13C are diagrams illustrating an example of mel-cepstral coefficients when the conversion device according to the second embodiment performs batch conversion in the inference phase and an example of mel-cepstral coefficients when the conversion device performs real-time conversion. [Figure 15] FIG. 11 is a diagram showing an example of mel-cepstral coefficients when the conversion device according to the first embodiment and the conversion device according to the second embodiment convert without taking into account either the context-preserving loss or the orthogonal attention loss, when converting while taking into account only the orthogonal attention loss, and when converting while taking into account only the context-preserving loss. [Figure 16] 5A to 5C are diagrams illustrating examples of mel-cepstral coefficients of a conventional conversion device, mel-cepstral coefficients of a conversion device according to the first embodiment, and mel-cepstral coefficients of a conversion device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] [First embodiment] An example of a transformation learning device and a transformation device according to the first embodiment will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is a diagram showing an example of a transformation learning device according to the first embodiment. As shown in Fig. 1, the transformation learning device 1 includes a transformation device 10 and a learning execution unit 20. The transformation device 10 also includes a source coding unit 11, a target coding unit 12, an attention matrix calculation unit 13, a target decoding unit 14, a target reconstruction unit 15, and a forward attention correction execution unit 16.

[0018] Fig. 2 is a diagram for explaining repetition of processes executed by a target encoding unit, an attention matrix calculation unit, and a target decoding unit according to the first embodiment. As shown in Fig. 1 and Fig. 2, the processes executed in processing block P, i.e., a series of processes executed by the target encoding unit 12, the attention matrix calculation unit 13, and the target decoding unit 14, are repeated L times (L: natural number).

[0019] The source encoding unit 11 converts a source domain feature quantity sequence X, which is a feature of the content data to be converted, into a first internal representation vector sequence Z using a first machine learning model.

[0020] The source domain is data that indicates the features of the content data to be converted, for example, the prosodic features of the speech data to be converted. The feature sequence X of the source domain is a feature vector x1, ..., x N , feature vector x1,…,x N The position is expressed by the following formula (1) including the dimension D of the content data and the number N of elements of the feature amount sequence of the source domain. In addition, when the content data is audio data, the position is a position on the time axis, that is, a time.

[0021]

number

[0022] The first machine learning model is, for example, a convolutional neural network (CNN). The first internal representation vector sequence Z is a matrix in which the internal representation vectors at each position of the source domain feature sequence are arranged, and is expressed by the following formulas (2) and (3). Formula (2) also includes the number of channels, that is, the dimension D of the internal representation vector. Z Contains:

[0023]

number

[0024]

number

[0025] The target encoding unit 12 converts the features of the conversion target content data, the feature quantity sequence Y of the target domain, into a second internal representation vector sequence Q using a second machine learning model. In addition, the speech sentence included in the conversion target content data is identical to the speech sentence included in the conversion source content data.

[0026] The target domain is data that indicates the prosodic features of the speech data to be converted. The feature sequence Y of the target domain is a feature vector y1, ..., y M , and the number of elements in the feature sequence of the target domain, M, is expressed as the following equation (4).

[0027]

number

[0028] The second machine learning model is, for example, a convolutional neural network. The second internal representation vector sequence Q is a matrix in which the internal representation vectors at each position of the feature sequence of the target domain are arranged, and is expressed by the following formulas (5) and (6). Formula (5) also includes the number of channels, that is, the dimension D of the internal representation vector. Q Contains:

[0029]

number

[0030]

number

[0031] The attention matrix calculation unit 13 calculates the attention matrix A using the first internal representation vector sequence Z and the second internal representation vector sequence Q. Then, the attention matrix calculation unit 13 calculates a third internal representation vector sequence R which is the product of the attention matrix A and an internal representation vector sequence V calculated by linearly transforming the first internal representation vector sequence Z.

[0032] First, the attention matrix calculation unit 13 performs a linear transformation L on the internal representation vector sequence Z of the source domain to calculate the internal representation vector sequence K and the internal representation vector sequence V, as shown in the following equation (7).

[0033]

number

[0034] Then, the attention matrix calculation unit 13 calculates the attention matrix A using the following formula (8) including the internal representation vector sequence K and the second internal representation vector sequence Q. Note that "softmax" included in formula (8) means applying a softmax function to each column of the matrix. Also, D' included in formula (8) is the dimension of the column vectors included in the internal representation vector sequence K, the internal representation vector sequence Q, and the second internal representation vector sequence Q.

[0035]

number

[0036] The attention matrix A corresponds to a similarity matrix between the second internal representation vector sequence Q and the internal representation vector sequence K, and is a matrix that associates each position of the source domain feature sequence X with each position of the target domain feature sequence Y. Therefore, the third internal representation vector sequence R, which is a matrix obtained by stretching and shortening the internal representation vector sequence V in accordance with the time axis of the target domain feature sequence Y, is expressed by the following formula (9).

[0037]

number

[0038] Alternatively, the attention matrix calculation unit 13 performs a linear transformation L Z Then, the second internal representation vector sequence Q is linearly transformed into Q Then, H sets of internal representation vector sequences K h , internal representation vector sequence V h and the second internal representation vector sequence Q h Here, "split" included in formula (12) and formula (13) represents division in the channel direction.

[0039]

number

[0040]

number

[0041]

number

[0042]

number

[0043] Then, the attention matrix calculation unit 13 calculates the internal representation vector sequence K h and the second internal representation vector sequence Q h The attention matrix A is calculated using the following equation (14), which includes:

[0044]

number

[0045] In this case, the attention matrix calculation unit 13 calculates the third internal representation vector sequence R expressed by the following formulas (15) and (16). Note that "concat" included in formula (16) represents concatenation in the channel direction.

[0046]

number

[0047]

number

[0048] In the following description, the process in which the attention matrix calculation unit 13 calculates the third internal representation vector sequence R from the first internal representation vector sequence Z and the second internal representation vector sequence Q is represented as the following equation (17).

[0049]

number

[0050] The target decoding unit 14 calculates a feature sequence I of a transformation domain used to transform the source domain to the transformation domain using the third internal representation vector sequence R by using a third machine learning model, as shown in the following formula (18). The third machine learning model is, for example, a convolutional neural network.

[0051]

number

[0052] Here, in the learning phase in which the above-mentioned first machine learning model, second machine learning model, and third machine learning model are trained, a set of source domain and target domain is given. On the other hand, in the inference phase in which voice conversion is performed using these three machine learning models, only the source domain is given. Therefore, in the inference phase, for example, the feature sequence I of the conversion domain most recently output by the target encoding unit 12 is input.

[0053] For this reason, in the learning phase in which the first machine learning model, the second machine learning model, and the third machine learning model are trained, it is necessary to make the feature sequence I of the transformation domain match the time-shifted sequence of the feature sequence Y of the target domain as closely as possible. Furthermore, when calculating each output, the target encoding unit 12 and the target decoding unit 14 need to execute the above process under a condition in which the output after the output is calculated cannot be used. This corresponds to imposing causality on the convolution layer when the second machine learning model included in the target encoding unit 12 and the third machine learning model included in the target decoding unit 14 are realized by a convolutional neural network.

[0054] Therefore, the distance between the submatrix of the feature sequence I in the transform domain and the submatrix of the feature sequence Y in the target domain, expressed by the following equation (19), can be considered as the learning loss. Equation (19) includes the symbol || ||, which indicates the norm of the matrix. Equation (19) also expresses the submatrix I consisting of elements from row d to row d' and column m to column m' of the feature sequence I in the transform domain. d:d´,m:m´ and a submatrix Y consisting of elements from row d to row d' and column m to column m' of the feature sequence Y of the target domain. d:d´,m:m´ Contains:

[0055]

number

[0056] The learning execution unit 20 trains at least one of the target encoding unit 12 and the target decoding unit 14 so as to shorten the distance between the feature sequence Y of the target domain and the feature sequence I of the transformation domain. For example, the learning execution unit 20 trains at least one of the target encoding unit 12 and the target decoding unit 14 so as to make the value of the right-hand side of equation (19) as small as possible.

[0057] By taking into account the learning loss shown by equation (19), each element vector can include information on the feature amount of the target domain at a position after the current position in the third internal representation vector sequence R. Also, by taking into account the learning loss shown by equation (19), it is possible to combine information on the feature amount of the target domain at the same position as the current position, and to stabilize the inference result in the inference phase. Therefore, the target reconstruction unit 15 is introduced.

[0058] The target reconstruction unit 15 calculates, by a fourth machine learning model, a feature sequence J of a reconstructed target domain, which is a domain obtained by reconstructing the target domain using the third internal representation vector sequence R, as shown in the following formula (20). The fourth machine learning model is, for example, a convolutional neural network.

[0059]

number

[0060] The learning execution unit 20 trains the target reconstruction unit 15 so as to shorten the distance between the feature sequence Y of the target domain and the feature sequence J of the reconstructed target domain. For example, the learning execution unit 20 trains the target reconstruction unit 15 so as to make the value of the right side of the following equation (21) as small as possible. Moreover, the learning loss expressed by equation (21) is called a context preservation loss (CPL).

[0061]

number

[0062] 1 and 2 is 2 or more, L third internal representation vector sequences R are calculated by the attention matrix calculation unit 13. In this case, the context retention loss is the arithmetic average of Equation (21) for at least a part of the L third internal representation vector sequences R.

[0063] Furthermore, as described above, since the utterance sentences contained in the source content data and the utterance sentences contained in the target content data are identical, the attention matrix A often tends to have diagonal components and their neighboring elements dominating.

[0064] Therefore, the learning execution unit 20 trains at least one of the target encoding unit 12 and the target decoding unit 14 so as to reduce the off-diagonal components of the attention matrix A. For example, the learning execution unit 20 trains at least one of the target encoding unit 12 and the target decoding unit 14 so as to reduce the value of the right-hand side of the following equation (22) as much as possible. Moreover, the learning loss expressed by equation (22) is called Diagonal Attention Loss (DAL).

[0065]

number

[0066] However, in equation (22), W N×M The symbol between (ν) and A is the matrix W N×M It represents the element-wise product of (ν) and the attention matrix A. Also, the matrix W N×M (ν) is a matrix with N rows and M columns as shown in the following equation (23), and the (n, m)-th element is a matrix expressed by the following equation (24).

[0067]

number

[0068]

number

[0069] In addition, each position of the feature sequence Y of the target domain is considered to correspond to one or at most a few positions of the feature sequence X of the source domain. This means that any column vectors included in the attention matrix A are mutually orthogonal.

[0070] Therefore, the learning execution unit 20 trains at least one of the target encoding unit 12 and the target decoding unit 14 so as to increase the number of mutually orthogonal column vectors included in the attention matrix A. For example, the learning execution unit 20 trains at least one of the target encoding unit 12 and the target decoding unit 14 so as to make the value of the right-hand side of the following equation (25) as small as possible. Furthermore, the learning loss expressed by equation (25) is called orthogonal attention loss (OAL). However, in equation (25), W N×M (ρ) and (AA T ) and the symbol between them is the matrix W N×M (ρ) and (AA T )

[0071]

number

[0072] Considering the above equations (19), (21), (22), and (25), the learning loss is expressed by the following equation (26).

[0073]

number

[0074] In addition, in the inference phase, the target encoding unit 12 receives the transformed domain feature sequence I output by the target decoding unit 14, instead of the target domain feature sequence Y. Then, the target encoding unit 12 converts the transformed domain feature sequence I into a second internal representation vector sequence Q using a second machine learning model.

[0075] In addition, when the inference phase is in progress and the number of repetitions L of the process executed in the processing block P shown in Figures 1 and 2 is 2 or more, the forward attention correction execution unit 16 restricts the attention matrix calculation unit 13 so that the peak of the distribution of the attention positions of the attention matrix A changes monotonically and continuously. When the content data is audio data, the attention position referred to here is the attention time. Note that the conversion device 10 does not necessarily have to include the forward attention correction execution unit 16.

[0076] Furthermore, when the conversion device 10 performs conversion in real time in the inference phase, the learning execution unit 20 trains the source coding unit 11 under the constraint that the source domain feature sequence X input to the source coding unit 11 before the position at which the source domain feature sequence X is converted into the first internal representation vector sequence Z is converted into the first internal representation vector sequence Z.

[0077] Next, an example of a process executed by the transformation learning device 1 in the learning phase will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of a process executed by the transformation learning device according to the first embodiment in the learning phase. Note that the process shown in Fig. 3 is executed every time a pair of a source domain and a target domain is input to the transformation learning device 1.

[0078] In step S31, the source encoding unit 11 converts the feature quantity sequence X of the source domain into a first internal representation vector sequence Z.

[0079] In step S32, the target encoding unit 12 converts the feature quantity sequence Y of the target domain into a second internal representation vector sequence Q.

[0080] In step S33, the attention matrix calculation unit 13 calculates the attention matrix A by using the first internal representation vector sequence Z and the second internal representation vector sequence Q.

[0081] In step S34, the attention matrix calculation unit 13 calculates a third internal representation vector sequence R which is the product of the attention matrix A and the internal representation vector sequence V calculated by linearly transforming the first internal representation vector sequence Z.

[0082] In step S35, the target decoding unit 14 uses the third internal representation vector sequence R to calculate a transform domain feature sequence I that is used to transform the source domain into the transform domain.

[0083] In step S36, the target reconstruction unit 15 uses the third internal representation vector sequence R to calculate a feature amount sequence J of a reconstructed target domain, which is a domain obtained by reconstructing the target domain.

[0084] In step S37, the learning execution unit 20 trains at least one of the source encoding unit, the target encoding unit, the target decoding unit, and the target reconstruction unit.

[0085] Note that the conversion learning device 1 may switch the order of steps S31 and S32. Also, the conversion learning device 1 may switch the order of steps S35 and S36.

[0086] Next, an example of processing executed by the conversion device 10 in the inference phase will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of processing executed by the conversion device according to the first embodiment in the inference phase.

[0087] In step S41, the source encoding unit 11 converts the feature quantity sequence X of the source domain into a first internal representation vector sequence Z.

[0088] In step S42, the target encoding unit 12 converts the feature quantity sequence Y of the target domain into a second internal representation vector sequence Q.

[0089] In step S43, the attention matrix calculation unit 13 calculates the attention matrix A by using the first internal representation vector sequence Z and the second internal representation vector sequence Q.

[0090] In step S44, the attention matrix calculation unit 13 calculates a third internal representation vector sequence R which is the product of the attention matrix A and the internal representation vector sequence V calculated by linearly transforming the first internal representation vector sequence Z.

[0091] In step S45, the target decoding unit 14 uses the third internal representation vector sequence R to calculate a transform domain feature sequence I that is used to transform the source domain into the transform domain.

[0092] Conversion device 10 may switch the order of steps S41 and S42.

[0093] The above describes the transformation learning device 1 and the transformation device 10 according to the first embodiment. The transformation learning device 1 transforms the feature sequence X of the source domain into a first internal representation vector sequence Z using a first machine learning model, and transforms the feature sequence Y of the target domain into a second internal representation vector sequence Q using a second machine learning model. Next, the transformation learning device 1 calculates an attention matrix A using the first internal representation vector sequence X and the second internal representation vector sequence Y, and calculates a third internal representation vector sequence R, which is a product of the internal representation vector sequence V, which is calculated by linearly transforming the first internal representation vector sequence X, and the attention matrix A. Then, the transformation learning device 1 calculates a feature sequence I of the transformation domain, which is used to transform the source domain into the transformation domain, using the third internal representation vector sequence R, by the third machine learning model. In addition, the transformation learning device 1 trains at least one of the target encoding unit 12 and the target decoding unit 14 so that the distance between the submatrix of the feature sequence Y of the target domain and the submatrix of the feature sequence I of the transformation domain is shortened.

[0094] As a result, the conversion learning device 1 can execute highly accurate learning with a higher probability, generate an effective learning model, and enable execution of an effective conversion process using the learning model. Furthermore, when the above-mentioned content data is speech data, the conversion learning device 1 can enable execution of an effective conversion process including prosodic features.

[0095] Furthermore, the transformation learning device 1 calculates, by a fourth machine learning model, a feature sequence I of a reconstructed target domain, which is a domain obtained by reconstructing the target domain using the third internal representation vector sequence R. Then, the transformation learning device 1 trains the target reconstruction unit so as to shorten the distance between the feature sequence Y of the target domain and the feature sequence J of the reconstructed target domain. This enables the transformation learning device 1 to enable the conversion device 10 to output a more stable inference result.

[0096] Furthermore, the conversion learning device 1 trains at least one of the target encoding unit 12 and the target decoding unit 14 so as to reduce the off-diagonal components of the attention matrix A. This enables the conversion learning device 1 to perform learning on the premise that the spoken sentence of the content data including the source domain matches the spoken sentence of the content data including the target domain, and enables the conversion device 10 to output a more stable inference result.

[0097] Furthermore, the transformation learning device 1 trains at least one of the target encoding unit 12 and the target decoding unit 14 so as to increase the number of mutually orthogonal column vectors included in the attention matrix A. This enables the transformation learning device 1 to perform learning on the premise that each position of the feature sequence X in the source domain roughly corresponds to each position of the feature sequence Y in the target domain, and enables the conversion device 10 to output a more stable inference result.

[0098] Furthermore, the transformation learning device 1 trains the source coding unit 11 under the constraint that the source domain feature sequence X input to the source coding unit 11 before the position where the source domain feature sequence X is converted into the first internal representation vector sequence Z is converted into the first internal representation vector sequence Z. In this way, the transformation learning device 1 can enable the conversion device 10 to perform the conversion process online in real time.

[0099] The conversion device 10 performs the conversion process using the feature quantity sequence of the conversion domain by the conversion learning device 1, and therefore can perform an effective conversion process.

[0100] Furthermore, when the process of the target encoding unit 12 converting the feature sequence I of the transformation domain into the second internal representation vector sequence Q, the attention matrix calculation unit 13 calculating the third internal representation vector sequence R, and the target decoding unit 14 calculating the feature sequence I of the transformation domain is executed at least twice, the conversion device 10 restricts the attention matrix calculation unit so that the peak of the distribution of attention positions of the attention matrix A changes monotonically and continuously. This allows the conversion device 10 to execute the conversion process while making the peak of the distribution of attention positions progress continuously.

[0101] Next, specific examples of effects achieved by the transformation learning device 1 and the transformation device 10 according to the first embodiment will be described with reference to Fig. 5 to Fig. 7. In the description of the effects, the source encoding unit 11, the target encoding unit 12, the target decoding unit 14, and the target reconstruction unit 15 all include convolutional layers.

[0102] The acoustic feature vector is a combination of the mel-cepstral coefficient, the logarithmic fundamental frequency pattern, the aperiodicity index, and the voiced / unvoiced indicator, which is set to "1" for voiced segments and "0" for unvoiced segments.

[0103] Furthermore, Mel-Cepstral Distortion (MCD) was used as an evaluation index. If the Mel-Cepstral coefficients of the target domain and the Mel-Cepstral coefficients of the transform domain are expressed by Equation (27) and Equation (28), respectively, the Mel-Cepstral Distortion is expressed as Equation (29).

[0104]

number

[0105]

number

[0106]

number

[0107] However, since the target domain and the transform domain are not necessarily time-aligned, the corresponding positions of the two series are determined using dynamic time warping (DTW) with the Mel-cepstral coefficients as the local distance, and the quality of the transform domain is evaluated by calculating the average of the Mel-cepstral coefficients at each corresponding position.

[0108] Fig. 5 is a diagram showing an example of an attention matrix when the conversion device according to the first embodiment does not perform forward attention correction in the inference phase. In Fig. 5, the horizontal axis indicates the time corresponding to each feature vector included in the target domain, and the vertical axis indicates the time corresponding to each feature vector included in the source domain. Referring to Fig. 5, it can be seen that the peak of the distribution of attention positions of attention matrix A is interrupted even though the spoken sentences are aligned in the source domain and the target domain. This is because forward attention correction is not performed.

[0109] Fig. 6 is a diagram showing an example of an attention matrix when the conversion device according to the first embodiment performs forward attention correction in the inference phase. In Fig. 6, the horizontal axis indicates the time corresponding to each feature vector included in the target domain, and the vertical axis indicates the time corresponding to each feature vector included in the source domain. Referring to Fig. 6, it can be seen that the peak of the distribution of attention positions of the attention matrix A changes monotonically and continuously. This is because forward attention correction is being performed.

[0110] 7 is a diagram showing an example of the mel-cepstral coefficients when the conversion device according to the first embodiment does not perform forward attention correction and when it performs forward attention correction. Referring to the table shown in FIG. 7, it can be seen that the mel-cepstral distortion is smaller when forward attention correction is performed than when forward attention correction is not performed, and the conversion is performed effectively.

[0111] [Second embodiment] An example of a transformation learning device and a conversion device according to the second embodiment will be described with reference to FIG. 8. FIG. 8 is a diagram showing an example of a transformation learning device according to the second embodiment. As shown in FIG. 8, the transformation learning device 3 includes a transformation device 30 and a learning execution unit 40. The transformation device 30 also includes a source coding unit 31, a target coding unit 32, an attention matrix calculation unit 33, a target decoding unit 34, and a target reconstruction unit 35. Unlike the transformation learning device 1 and the conversion device 10 according to the first embodiment, the transformation learning device 3 and the conversion device 30 are capable of decoding a feature sequence X of the same speech voice belonging to a plurality of domains, for example, K types of domains, in a single network set. (1) ,…,X (K) It allows conversion between

[0112] The source encoding unit 31 encodes a feature sequence X (k) First, use a machine learning model to create the first internal representation of the vector sequence Z (k) The first internal representation vector sequence Z (k)is a matrix of internal representation vectors at each position of the feature sequence of the first domain. In addition, when the content data is audio data, the position is a position on the time axis, that is, a time. (k) and the first internal representation vector sequence Z (k) The relationship between is expressed by the following formula (30): The first machine learning model is, for example, a convolutional neural network.

[0113]

number

[0114] The target encoding unit 32 encodes a feature quantity sequence X of a second domain, which is a feature of the content data to be converted and has a second index k′ different from the first index k. (k´) The second machine learning model is used to represent the second internal vector sequence Q (k´) The second internal representation vector sequence Q (k´) is a matrix that lists the internal representation vectors at each position of the feature sequence of the second domain. (k´) and the second internal representation vector sequence Q (k´) The relationship between is expressed by the following formula (31): The second machine learning model is, for example, a convolutional neural network.

[0115]

number

[0116] First, the attention matrix calculation unit 33, like the attention matrix calculation unit 13 according to the first embodiment, calculates the first internal representation vector sequence Z (k) and the second internal representation vector sequence Q (k´) Using the following equations (32) and (33), we obtain the attention matrix A (k,k´) Calculate.

[0117]

number

[0118]

number

[0119] Then, like the attention matrix calculation unit 13 according to the first embodiment, the attention matrix calculation unit 33 calculates the third internal representation vector sequence R by using the following equation (34). (k,k´) Calculate.

[0120]

number

[0121] Alternatively, the attention matrix calculation unit 33, like the attention matrix calculation unit 13 according to the first embodiment, calculates the internal representation vector sequence Z (k) Linear transformation to L Z Then, the second internal representation vector sequence Q (k´) Linear transformation to L Q Then, H sets of internal representation vector sequences K h (k) , internal representation vector sequence V h (k) and the second internal representation vector sequence Q (k´) Calculate.

[0122]

number

[0123]

number

[0124]

number

[0125]

number

[0126] Then, the attention matrix calculation unit 33, like the attention matrix calculation unit 13 according to the first embodiment, calculates the internal representation vector sequence K h (k) and the second internal representation vector sequence Q h (k´) Using the following equation (39), we obtain the attention matrix A h (k,k´) Calculate.

[0127]

number

[0128] In this case, the attention matrix calculation unit 33 calculates the third internal representation vector sequence R (k,k´) Calculate.

[0129]

number

[0130]

number

[0131] In the following description, the attention matrix calculation unit 33 calculates the first internal representation vector sequence Z (k) and the second internal representation vector sequence Q (k´) from the third internal representation vector sequence R (k,k´) The process of calculating is represented as the following equation (42).

[0132]

number

[0133] The target decoding unit 34 decodes the third internal representation vector sequence R (k,k´) The transformed domain feature sequence F is used to transform the first domain into the transformed domain using a third machine learning model.(k,k´) The third machine learning model is, for example, a convolutional neural network.

[0134]

number

[0135] Here, in the learning phase in which the above-mentioned first machine learning model, second machine learning model, and third machine learning model are trained, a set of a first domain and a target domain is given. On the other hand, in the inference phase in which voice conversion is performed using these three machine learning models, only the first domain is given. Therefore, in the inference phase, the target encoding unit 32, for example, uses the feature sequence F of the conversion domain most recently output by itself as (k,k´) is entered.

[0136] Feature sequence F in the transform domain (k,k´) and the feature sequence of the second domain X (k´) The learning loss expressed as the distance between is expressed by the following equation (44). Equation (44) also expresses the feature sequence F (k,k´) The submatrix F consisting of elements from row d to row d' and column m to column m' of (k,k´) d:d´,m:m´ and the feature sequence of the second domain X (k´) The submatrix X consisting of elements from row d to row d' and column m to column m' of (k´) d:d´,m:m´ Contains:

[0137]

number

[0138] The learning execution unit 40 learns the feature sequence X (k´) and the feature sequence F in the transform domain (k,k´)At least one of the target encoding unit 32 and the target decoding unit 34 is trained so as to shorten the distance between the submatrix of (44) and the submatrix of (45). For example, the training execution unit 40 trains at least one of the target encoding unit 32 and the target decoding unit 34 so as to make the value of the right-hand side of equation (44) as small as possible.

[0139] By taking into account the learning loss shown in equation (44), the third internal representation vector sequence R (k,k´) Each element vector can include information on the feature quantity of the second domain at a position after the current position. Also, by taking into account the learning loss shown in equation (44), the feature quantity of the second domain at the same position as the current position can be included, and the inference result in the inference phase can be stabilized. Therefore, the target reconstruction unit 35 is introduced.

[0140] The target reconstruction unit 35 calculates the fourth internal representation vector sequence R (k,k´) The feature sequence J of the reconstructed domain is the domain in which the second domain is reconstructed using (k,k´) is calculated by a fourth machine learning model. The fourth machine learning model is, for example, a convolutional neural network.

[0141]

number

[0142] The learning execution unit 40 learns the feature sequence X (k´) and the feature sequence of the reconstructed domain J (k,k´) The target reconstruction unit 35 is trained so that the distance between is shortened. For example, the learning execution unit 40 trains the target reconstruction unit 15 so that the value of the right side of the following equation (46) is as small as possible. The learning loss expressed by equation (46) is called a context-preserving loss.

[0143]

number

[0144] 8 is 2 or more, L third internal representation vector sequences R are calculated by the attention matrix calculation unit 33. In this case, the context retention loss is the arithmetic average of Equation (46) for at least a part of the L third internal representation vector sequences R.

[0145] In addition, as in the first embodiment, the utterance sentences included in the content data of the conversion source and the utterance sentences included in the content data of the conversion target are the same, so the attention matrix A (k,k´) often tends to be dominated by diagonal elements and their neighbors.

[0146] Therefore, the learning execution unit 40 calculates the attention matrix A (k,k´) At least one of the target encoding unit 32 and the target decoding unit 34 is trained so that the off-diagonal components of are reduced. For example, the learning execution unit 40 trains at least one of the target encoding unit 12 and the target decoding unit 14 so that the value of the right-hand side of the following equation (47) is as small as possible. The learning loss expressed by equation (47) is called a diagonal attention loss.

[0147]

number

[0148] Similarly to the first embodiment, the feature sequence X (k´) Each position of is the feature sequence of the first domain X (k) This is thought to be one or at most a few of the positions in the attention matrix A (k,k´) This means that any column vectors in are orthogonal to each other.

[0149] Therefore, the learning execution unit 40 calculates the attention matrix A (k,k´)At least one of the target encoding unit 32 and the target decoding unit 34 is trained so that the number of mutually orthogonal column vectors included in is increased. For example, the learning execution unit 40 trains at least one of the target encoding unit 32 and the target decoding unit 34 so that the value of the right-hand side of the following equation (48) is as small as possible. Moreover, the learning loss expressed by equation (48) is called orthogonal attention loss.

[0150]

number

[0151] The above formulas (44), (46), (47) and (48) ), the learning loss is expressed by the following equation (49).

[0152]

number

[0153] In addition, in the inference phase, the target encoding unit 32 encodes the feature sequence X (k´) Instead, the transformed domain feature sequence F (k,k´) Then, the target encoding unit 32 uses the second machine learning model to encode the transformed domain feature sequence F (k,k´) The second internal representation vector sequence Q (k´) Convert to.

[0154] Next, the layers included in the networks included in the conversion learning device 3 and the conversion device 30 will be described using equations (50) to (57) as appropriate.

[0155] The networks included in the conversion learning device 3 and the conversion device 30 preferably include a convolution layer and a self-attention layer for the following reasons. In the conversion learning device 3 and the conversion device 30, the input distribution to each hidden layer of each network may change significantly depending on at least one of the first domain and the second domain. In addition, even if a batch normalization layer, an instance normalization layer, etc. are inserted as usual, the range of values ​​that each layer should cover is inevitably wider than that of the conversion learning device 1 and the conversion device 10 according to the first embodiment, so that the accuracy of the inference result may decrease in the inference phase.

[0156] In order to avoid such a situation, it is preferable that the networks included in the conversion learning device 3 and the conversion device 30 include a layer that performs normalization, weight normalization, or layer normalization according to the first domain or the second domain. That is, it is preferable that the source coding unit 31 has a neural network including a layer that performs normalization, weight normalization, or layer normalization according to the first domain. Furthermore, it is preferable that the target coding unit 32 has a neural network including a layer that performs normalization, weight normalization, or layer normalization according to the second domain. Moreover, it is preferable that the target decoding unit 34 has a neural network including a layer that performs normalization, weight normalization, or layer normalization according to the first domain. Moreover, it is preferable that the target reconstruction unit 35 has a neural network including a layer that performs normalization, weight normalization, or layer normalization according to the second domain.

[0157] For example, a typical batch normalization process is expressed by the following equation (47).

[0158]

number

[0159] The batch normalization represented by the formula (47) corresponds to performing the operation represented by the following formula (51) for each element. Formula (51) expresses the average μ d (X), standard deviation σ of the dth channel d (X) and standard deviation σ d (X). Equation (51) is expressed as equation (52), and the standard deviation σ d The shift amount γ corresponding to (X) and the batch average μ d It includes a scaling factor β corresponding to (X).

[0160]

number

[0161]

number

[0162]

number

[0163] Here, when the idea of ​​normalization according to the first domain or the second domain is applied to batch normalization, the batch normalization depends on the index and is therefore expressed by the following equation (54).

[0164]

number

[0165] The batch normalization represented by the formula (54) corresponds to performing the operation represented by the following formula (55) for each element. Formula (55) expresses the average μ d (X (k) ), the standard deviation of the dth channel σ d (X (k) ) and standard deviation σ d (X (k)) is included. Equation (56) is expressed as equation (57), and the standard deviation σ d (X (k) ) the shift amount γ k and (53), the batch average μ d (X (k) ) corresponding scaling factor β k Contains:

[0166]

number

[0167]

number

[0168]

number

[0169] Note that the networks included in the transformation learning device 3 and the transformation device 30 may include layers that perform weight normalization or layer normalization, as described above.

[0170] In the above description, it is assumed that the first domain input to the transformation learning device 3 and the conversion device 30 is known, but the first domain may be unknown. In such a case, the transformation learning device 3 and the conversion device 30 become devices that perform many-to-one conversion, and the source coding unit 31 converts the feature sequence X (k) Only.

[0171]

number

[0172] In addition, when the conversion device 30 performs conversion in real time in the inference phase, the learning execution unit 40 converts the feature quantity sequence X (k) is the first internal representation vector sequence Z (k)The feature sequence X of the first domain input to the source coding unit 31 before the position where the feature sequence X (k) The first internal representation of the vector sequence Z (k) The source coding unit 31 is trained under the constraint of transforming

[0173] Next, an example of a process executed by the conversion learning device 3 in the learning phase will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of a process executed by the conversion learning device according to the second embodiment in the learning phase. Note that the process shown in Fig. 9 is executed every time a pair of a first domain and a second domain is input to the conversion learning device 3.

[0174] In step S91, the source coding unit 31 converts the feature quantity sequence X (k) The first internal representation of the vector sequence Z (k) Convert to.

[0175] In step S92, the target encoding unit 32 encodes the feature sequence X (k´) The second internal representation vector sequence Q (k´) Convert to.

[0176] In step S93, the attention matrix calculation unit 33 calculates the first internal representation vector sequence Z (k) and the second internal representation vector sequence Q (k´) Using the attention matrix A (k,k´) Calculate.

[0177] In step S94, the attention matrix calculation unit 33 calculates the first internal representation vector sequence Z (k) The internal representation vector sequence V is calculated by linearly transforming h (k) and attention matrix A (k,k´) The third internal representation vector sequence R (k,k´) Calculate.

[0178] In step S95, the target decoding unit 34 decodes the third internal representation vector sequence R (k,k´)The transform domain feature sequence F is used to transform the first domain to the transform domain using (k,k´) Calculate.

[0179] In step S96, the target reconstructing unit 35 calculates the third internal representation vector sequence R (k,k´) The feature sequence J of the reconstructed domain is the domain in which the second domain is reconstructed using (k,k´) Calculate.

[0180] In step S97, the learning execution unit 40 trains at least one of the source encoding unit 31, the target encoding unit 32, the target decoding unit 34, and the target reconstruction unit 35.

[0181] The conversion learning device 3 may switch the order of steps S91 and S92. Also, the conversion learning device 3 may switch the order of steps S95 and S96.

[0182] Next, an example of processing executed by the conversion device 30 in the inference phase will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of processing executed by the conversion device according to the second embodiment in the inference phase.

[0183] In step S101, the source coding unit 31 converts the feature quantity sequence X (k) The first internal representation of the vector sequence Z (k) Convert to.

[0184] In step S102, the target encoding unit 32 encodes the feature sequence X (k´) The second internal representation vector sequence Q (k´) Convert to.

[0185] In step S103, the attention matrix calculation unit 33 calculates the first internal representation vector sequence Z (k) and the second internal representation vector sequence Q (k´) Using the attention matrix A (k,k´) Calculate.

[0186] In step S104, the attention matrix calculation unit 33 calculates the first internal representation vector sequence Z (k) The internal representation vector sequence V is calculated by linearly transforming h (k) and attention matrix A (k,k´) The third internal representation vector sequence R (k,k´) Calculate.

[0187] In step S105, the target decoding unit 34 decodes the third internal representation vector sequence R (k,k´) The transform domain feature sequence F is used to transform the first domain to the transform domain using (k,k´) Calculate.

[0188] The conversion device 30 may switch the order of steps S101 and S102.

[0189] Next, the above-mentioned convolution layer will be described with reference to formulas (59) to (65). When the input is expressed by formula (59) and the output is expressed by formula (60), the convolution layer is expressed by formula (61) including a linear transformation L and formula (62) including a nonlinear transformation f. The nonlinear transformation f is, for example, a sigmoid function, a normalized linear function, or a gated linear function.

[0190]

number

[0191]

number

[0192]

number

[0193]

number

[0194] Furthermore, when there is a causality constraint, the linear transformation L is expressed by the following equation (63), which includes the convolution kernel size N', when expressed element by element.

[0195]

number

[0196] The parameters to be learned in this layer are the coefficient W of the linear transformation L expressed by the following equation (64) and the coefficient b expressed by the following equation (65).

[0197]

number

[0198]

number

[0199] Next, the above-mentioned self-attention layer will be explained with reference to equations (66) to (70). When the input is expressed by equation (59) and the output is expressed by equation (60), the self-attention layer is expressed by the following equations (66) to (68).

[0200]

number

[0201]

number

[0202]

number

[0203] Equations (63) to (68) represent the following processing. First, an internal representation vector sequence Q, an internal representation vector sequence K, and an internal representation vector sequence V expressed by the following equation (69) are generated by linear conversion from the number of channels D of the input X to the number of channels 3D', for example, by a convolution operation with a kernel size of 1. Next, a self-similarity matrix A between the internal representation vector sequence Q and the internal representation vector sequence K is calculated. Then, the output Y is calculated using equation (68).

[0204]

number

[0205] However, when causal constraints are not taken into account, the matrix M included in equation (67) is a matrix whose all elements are zero. When causal constraints are taken into account, the matrix M is a matrix whose (m,n)-th element is expressed by the following equation (70).

[0206]

number

[0207] Next, the above-mentioned multiple self-attention layer will be described with reference to equations (71) to (78). When the input is expressed by equation (59) and the output is expressed by equation (60), the multiple self-attention layer executes the operations expressed by the following equations (71) to (78). Equation (71) includes a linear transformation L1. Moreover, equation (78) includes a linear transformation L2.

[0208]

number

[0209]

number

[0210]

number

[0211]

number

[0212]

number

[0213]

number

[0214]

number

[0215]

number

[0216] The transformation learning device 3 and the conversion device 30 according to the second embodiment have been described above. The transformation learning device 3 converts a feature sequence X (k) First, use a machine learning model to create the first internal representation of the vector sequence Z (k) In addition, the transformation learning device 3 transforms the feature sequence X (k´) The second machine learning model is used to represent the second internal vector sequence Q (k´) Next, the transformation learning device 3 transforms the first internal representation vector sequence Z (k) and the second internal representation vector sequence Q (k´) Using the attention matrix A (k,k´) The transformation learning device 3 calculates the first internal representation vector sequence Z (k) The internal representation vector sequence V is calculated by linearly transforming (k) and attention matrix A (k,k´) The third internal representation vector sequence R (k,k´) Then, the transformation learning device 3 calculates the third internal representation vector sequence R(k,k´) The feature sequence I of the transform domain is used to transform the first domain to the transform domain using (k,k´) is calculated by the third machine learning model. In addition, the transformation learning device 3 calculates the feature sequence X (k) Submatrix of and feature sequence of transform domain I (k,k´) At least one of the target encoding unit 32 and the target decoding unit 34 is trained so as to shorten the distance between the submatrix of the target and the submatrix of the target.

[0217] This allows the conversion learning device 3 to generate a learning model that enables conversion between a plurality of domains with a single network set. This also allows the conversion learning device 3 to perform an effective conversion process using the learning model. Furthermore, when the above-mentioned content data is speech data, the conversion learning device 3 can perform an effective conversion process including prosodic features.

[0218] In addition, the transformation learning device 3 converts the third internal representation vector sequence R (k,k´) The feature sequence J of the reconstructed domain is the domain in which the second domain is reconstructed using (k,k´) is calculated by the fourth machine learning model. Then, the transformation learning device 3 calculates the feature sequence X (k´) and the feature sequence of the reconstructed domain J (k,k´) At least one of the target encoding unit 32 and the target decoding unit 34 is trained so as to shorten the distance between the target and the learning unit 3. This enables the conversion device 30 to output a more stable inference result.

[0219] In addition, the transformation learning device 3 uses the attention matrix A (k,k´) At least one of the target encoding unit 32 and the target decoding unit 34 is trained so that the off-diagonal components of are reduced. This enables the conversion learning device 3 to learn on the premise that the spoken sentence of the content data including the first domain matches the spoken sentence of the content data including the second domain, and enables the conversion device 30 to output a more stable inference result.

[0220] In addition, the transformation learning device 3 uses the attention matrix A (k,k´) At least one of the target encoding unit 32 and the target decoding unit 34 is trained so that the number of mutually orthogonal column vectors included in the (k) and the feature sequence X of the target domain (k´) This enables learning based on the assumption that the positions of the two roughly correspond to each other, and enables the conversion device 30 to output more stable inference results.

[0221] In addition, in the transformation learning device 3, the source coding unit 31 has a neural network including a layer that performs normalization, weight normalization, or layer normalization according to the first domain. In addition, the target coding unit 32 has a neural network including a layer that performs normalization, weight normalization, or layer normalization according to the second domain. Similarly, the target decoding unit 34 has a neural network including a layer that performs normalization, weight normalization, or layer normalization according to the first domain. In addition, the target reconstruction unit 35 has a neural network including a layer that performs normalization, weight normalization, or layer normalization according to the second domain. This enables the transformation learning device 3 to perform normalization, weight normalization, or layer normalization, and reduces the possibility that the accuracy of the inference result will decrease in the inference phase.

[0222] In addition, the transformation learning device 3 converts the feature sequence X (k) is the first internal representation vector sequence Z (k) The feature sequence X of the first domain input to the source coding unit 31 before the position where the feature sequence X (k) The first internal representation of the vector sequence Z (k) Thus, the conversion learning device 3 enables the conversion device 30 to execute the conversion process online in real time.

[0223] Next, specific examples of effects achieved by the transformation learning device 3 and the transformation device 30 according to the second embodiment will be described with reference to Fig. 11 to Fig. 16. In the description of the effects, the source encoding unit 31, the target encoding unit 32, the target decoding unit 34, and the target reconstruction unit 35 all include convolutional layers.

[0224] The acoustic feature vector is a combination of the mel-cepstral coefficient, the logarithmic fundamental frequency pattern, the aperiodicity index, and the voiced / unvoiced indicator, which is set to "1" for voiced segments and "0" for unvoiced segments.

[0225] Furthermore, Mel-Cepstral Distortion (MCD) was used as an evaluation index. If the Mel-Cepstral coefficients of the target domain and the Mel-Cepstral coefficients of the transform domain are respectively expressed by the above-mentioned formula (27) and formula (28), the Mel-Cepstral Distortion is expressed as the above-mentioned formula (29).

[0226] FIG. 11 is a diagram showing an example of an attention matrix in the case where the source coding unit, the target coding unit, the target decoding unit, and the target reconstruction unit according to the second embodiment have a normal batch normalization layer in the inference phase. In FIG. 11, the horizontal axis indicates the time corresponding to each feature vector included in the target domain, and the vertical axis indicates the time corresponding to each feature vector included in the source domain. Referring to FIG. 11, it can be seen that the peak of the distribution of the attention position gradually becomes unclear. This is because a normal batch normalization layer is used. Note that the attention position is the attention time when the content data is audio data.

[0227] 12 is a diagram showing an example of an attention matrix in the case where the source coding unit, the target coding unit, the target decoding unit, and the target reconstruction unit according to the second embodiment have a batch normalization layer corresponding to the first domain or the second domain in the inference phase. In FIG. 12, the horizontal axis indicates the time corresponding to each feature vector included in the target domain, and the vertical axis indicates the time corresponding to each feature vector included in the source domain. Referring to FIG. 12, it can be seen that the peak of the distribution of attention positions changes monotonically and continuously. This is because a batch normalization layer corresponding to the first domain or the second domain is used.

[0228] FIG. 13 is a diagram showing an example of mel-cepstral coefficients when the conversion device according to the second embodiment executes one-to-one conversion and many-to-one conversion in the inference phase. In FIG. 13, the third column from the left represents the mel-cepstral distortion when one-to-one conversion is executed, and the fourth column from the left represents the mel-cepstral distortion when many-to-one conversion is executed. Many-to-one conversion is advantageous in practical terms because it does not require input information, but the accuracy of conversion may decrease. In light of this, it can be seen that the conversion learning device 3 and the conversion device 30 can perform the same level of performance when executing many-to-one conversion as when executing one-to-one conversion.

[0229] Fig. 14 is a diagram showing an example of the mel-cepstral coefficients when the conversion device according to the second embodiment executes batch conversion in the inference phase and when the conversion device executes real-time conversion. In Fig. 14, the third column from the left represents the mel-cepstral distortion when batch processing is executed, and the fourth column from the left represents the mel-cepstral distortion when real-time processing is executed. Comparing the two, it can be seen that the conversion learning device 3 and the conversion device 30 can achieve the same level of performance when executing real-time processing as when executing batch processing.

[0230] FIG. 15 is a diagram showing examples of mel-cepstral coefficients when the conversion device according to the first embodiment and the conversion device according to the second embodiment perform conversion without taking into account either the context-preserving loss or the orthogonal attention loss, when the conversion device performs conversion while taking into account only the orthogonal attention loss, and when the conversion device performs conversion while taking into account only the context-preserving loss.

[0231] The third, fourth and fifth columns from the left in FIG. 15 show the mel cepstral coefficients of the conversion process by the conversion learning device 1 and the conversion device 10 according to the first embodiment. The sixth, seventh and eighth columns from the left in FIG. 15 show the mel cepstral coefficients of the conversion process by the conversion learning device 3 and the conversion device 30 according to the second embodiment. The third and sixth columns from the left in FIG. 15 show the mel cepstral coefficients when neither the context preserving loss nor the orthogonal loss is taken into account. The fourth and seventh columns from the left in FIG. 15 show the mel cepstral coefficients when only the context preserving loss is taken into account. The fifth and eighth columns from the left in FIG. 15 show the mel cepstral coefficients when only the context preserving loss is taken into account.

[0232] 15, it can be seen that in each of the conversion learning device 1 and the conversion device 10, and the conversion learning device 3 and the conversion device 30, the more learning losses are taken into consideration, the higher the conversion accuracy becomes.

[0233] FIG. 16 is a diagram showing an example of the mel cepstral coefficients of a conventional conversion device, the mel cepstral coefficients of a conversion device according to the first embodiment, and the mel cepstral coefficients of a conversion device according to the second embodiment. The third column from the left in FIG. 16 shows the mel cepstral coefficients of the conversion process by "sproket" which is a conventional technology. The fourth column from the left in FIG. 16 shows the mel cepstral coefficients of the conversion process by sequence conversion using a recurrent neural network (RNN). The fifth column from the left in FIG. 16 shows the mel cepstral coefficients of the conversion process by the conversion device 10 according to the first embodiment. The sixth column from the left in FIG. 16 shows the mel cepstral coefficients of the conversion process by the conversion device 30 according to the second embodiment.

[0234] Referring to FIG. 16, it can be seen that both the conversion device 10 and the conversion device 30 can perform conversion with higher accuracy than "sproket". Also, referring to FIG. 16, it can be seen that the conversion device 10 can perform conversion processing with the same accuracy as the conversion processing by sequence conversion using a recurrent neural network. Furthermore, considering that the conversion learning device using a recurrent neural network required about one week to learn, whereas the conversion device 10 and the conversion device 30 required about one day to learn, it can be seen that the conversion device 10 and the conversion device 30 have a higher effect than the conventional technology.

[0235] In the above-mentioned first and second embodiments, the content data is audio data, but the present invention is not limited thereto. The content data may be, for example, still image data or video data. However, when the content data is still image data, the above-mentioned position is a position on the spatial coordinates of the still image, for example, the position of each pixel included in the still image. Furthermore, when the content data is video data and a still image corresponding to a specific time of the video is focused on, the above-mentioned position is a position on the spatial coordinates of the still image. On the other hand, when the content data is video data and a specific position on the spatial coordinates, for example, a specific area including at least one pixel, is focused on, the above-mentioned position is a position on the time axis, that is, the time.

[0236] In addition, at least a part of the functions of the conversion learning device 1, the conversion device 10, the conversion learning device 3 and the conversion device 30 may be realized by hardware including circuitry such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit).

[0237] Furthermore, at least a part of the functions of the conversion learning device 1, the conversion device 10, the conversion learning device 3, and the conversion device 30 may be realized by cooperation between these hardware and software.

[0238] The software may be stored in, for example, a storage device having a non-transitory storage medium, and may be read and executed by the conversion learning device 1, the conversion device 10, the conversion learning device 3, and the conversion device 30. The storage device may be, for example, a hard disk drive (HDD: Hard Disk Drive) or a solid state drive (SSD: Solid State Drive).

[0239] Alternatively, the software may be stored in a storage device having a removable non-transitory storage medium, and may be read and executed by the conversion learning device 1, the conversion device 10, the conversion learning device 3, or the conversion device 30. The storage device may be, for example, a DVD or a CD-ROM.

[0240] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the embodiment of the present invention is not limited to the above-described embodiment, and at least one of various modifications, substitutions, and design changes may be made within the scope not departing from the gist of the present invention. [Explanation of symbols]

[0241] 1,3...transformation learning device, 10,30...transformation device, 11,31...source coding unit, 12,32...target coding unit, 13,33...attention matrix calculation unit, 14,34...target decoding unit, 15,35...target reconstruction unit, 16...forward attention correction execution unit, P,Q...processing block

Claims

1. a source encoding unit that converts a feature sequence of a first domain, which is a feature of source content data and has a first index, into a first internal representation vector sequence that is a matrix in which internal representation vectors at each position of the feature sequence of the first domain are arranged, using a first machine learning model; a target encoding unit that converts a feature sequence of a second domain, which is a feature of content data to be converted and is assigned a second index different from the first index, into a second internal representation vector sequence that is a matrix in which internal representation vectors at each position of the feature sequence of the second domain are arranged, using a second machine learning model; an attention matrix calculation unit that calculates an attention matrix, which is a matrix that associates each position of the feature sequence of the first domain with each position of the feature sequence of the second domain, using the first internal representation vector sequence and the second internal representation vector sequence, and calculates a third internal representation vector sequence that is a product of the attention matrix and an internal representation vector sequence calculated by linearly transforming the first internal representation vector sequence; a target decoding unit that calculates, by a third machine learning model, a feature sequence of the transformation domain that is used to transform the first domain into a transformation domain using the third internal representation vector sequence; a learning execution unit that trains at least one of the target encoding unit and the target decoding unit so that a distance between a submatrix of the feature sequence of the second domain and a submatrix of the feature sequence of the transformation domain becomes short with respect to the submatrix; Equipped with When the processing of the target encoding unit, the processing of the attention matrix calculation unit, and the processing of the target decoding unit are executed at least twice, the attention matrix calculation unit uses a batch normalization layer corresponding to the first domain to perform element-by-element normalization depending on the first index, or uses a batch normalization layer corresponding to the second domain to perform element-by-element normalization depending on the second index, thereby monotonically and continuously changing a peak of the distribution of attention positions of the attention matrix. Transformational learning device.

2. a target reconstructing unit that calculates, by a fourth machine learning model, a feature sequence of a reconstructed domain that is a domain obtained by reconstructing the second domain using the third internal representation vector sequence; the learning execution unit causes at least one of the target encoding unit and the target decoding unit to learn so as to shorten a distance between a feature sequence of the second domain and a feature sequence of the reconstructed domain; 2. The transformation learning device of claim 1.

3. The learning execution unit trains at least one of the target encoding unit and the target decoding unit so that off-diagonal elements of the attention matrix become smaller.

3. The conversion learning device according to claim 1 or 2.

4. A transformation learning method executed by a transformation learning device, comprising: a source encoding step of converting a feature sequence of a first domain, which is a feature of source content data and is assigned a first index, into a first internal representation vector sequence, which is a matrix in which internal representation vectors at each position of the feature sequence of the first domain are arranged, using a first machine learning model; a target encoding step of converting a feature sequence of a second domain, which is a feature of content data to be converted and is assigned a second index different from the first index, into a second internal representation vector sequence, which is a matrix in which internal representation vectors at each position of the feature sequence of the second domain are arranged, using a second machine learning model; an attention matrix calculation step of calculating an attention matrix, which is a matrix that associates each position of the feature sequence of the first domain with each position of the feature sequence of the second domain, using the first internal representation vector sequence and the second internal representation vector sequence, and calculating a third internal representation vector sequence that is a product of the attention matrix and an internal representation vector sequence calculated by linearly transforming the first internal representation vector sequence; a target decoding step of calculating, by a third machine learning model, a feature sequence of the transformation domain, which is used to transform the first domain into a transformation domain using the third internal representation vector sequence; a learning execution step of training at least one of a target encoding unit having the first machine learning model and a target decoding unit having the second machine learning model so that a distance between a submatrix of a feature sequence of the second domain and a submatrix of a feature sequence of the transformation domain becomes short with respect to a submatrix; Including, the attention matrix calculation step, when the target encoding step, the attention matrix calculation step, and the target decoding step are performed at least twice, includes: using a batch normalization layer corresponding to the first domain to perform element-by-element normalization depending on the first index; or using a batch normalization layer corresponding to the second domain to perform element-by-element normalization depending on the second index, thereby monotonically and continuously changing a peak of a distribution of attention positions of the attention matrix; Transformational learning methods.

5. A conversion learning program that causes a computer to function as the conversion learning device according to any one of claims 1 to 3.

6. a source encoding unit that converts a feature sequence of a first domain, which is a feature of source content data and has a first index, into a first internal representation vector sequence that is a matrix in which internal representation vectors at each position of the feature sequence of the first domain are arranged, using a first machine learning model; a target encoding unit that converts a feature sequence of a second domain, which is a feature of content data to be converted and is assigned a second index different from the first index, into a second internal representation vector sequence that is a matrix in which internal representation vectors at each position of the feature sequence of the second domain are arranged, using a second machine learning model; an attention matrix calculation unit that calculates an attention matrix, which is a matrix that associates each position of the feature sequence of the first domain with each position of the feature sequence of the second domain, using the first internal representation vector sequence and the second internal representation vector sequence, and calculates a third internal representation vector sequence that is a product of the attention matrix and an internal representation vector sequence calculated by linearly transforming the first internal representation vector sequence; a target decoding unit that calculates, by a third machine learning model, a feature sequence of the transformation domain that is used to transform the first domain into a transformation domain using the third internal representation vector sequence; Equipped with at least one of the target encoding unit and the target decoding unit is trained so as to shorten a distance between a submatrix of the feature sequence of the second domain and a submatrix of the feature sequence of the transformation domain; When the processing of the target encoding unit, the processing of the attention matrix calculation unit, and the processing of the target decoding unit are executed at least twice, the attention matrix calculation unit uses a batch normalization layer corresponding to the first domain to perform element-by-element normalization depending on the first index, or uses a batch normalization layer corresponding to the second domain to perform element-by-element normalization depending on the second index, thereby monotonically and continuously changing a peak of the distribution of attention positions of the attention matrix. Conversion device.

Images