Acoustic feature acquisition method and device, electronic equipment and readable storage medium

Abstract

Embodiments of the present application provide an acoustic feature acquisition method and device and a readable storage medium. The method comprises: acquiring a target phoneme sequence; performing encoding processing on the target phoneme sequence based on a split self-attention mechanism to obtain a first feature matrix; and performing decoding processing on the first feature matrix based on the split self-attention mechanism to obtain an acoustic feature of the target phoneme sequence. The split self-attention mechanism converts the unified processing multi-head attention in the multi-head self-attention mechanism into split multi-head attention by performing scale transformation on the key-value matrix and the value matrix in a preprocessing manner, realizes modeling of information at different levels and different scales at the same time, takes into account the overall and detailed grasp, improves the feature analysis capability of the phoneme feature, better meets the needs of emotional speech synthesis, and makes the expression of the finally synthesized speech more accurate and vivid.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and in particular to a method, apparatus, electronic device, and readable storage medium for acquiring acoustic features. Background Technology

[0002] Speech synthesis technology is a technique that transforms computer-generated or externally input text information into fluent, understandable spoken output. It is a cutting-edge technology in the field of Chinese information processing, primarily addressing the problem of converting text information into sound information. Speech synthesis technology generally consists of three parts: a front-end, an acoustic model, and a vocoder. The input to the acoustic model is a phoneme sequence converted from text by the front-end, and the output is the acoustic features of the speech. These acoustic features are then processed by the vocoder module to finally synthesize complete speech.

[0003] Existing classic speech synthesis models based on recurrent neural networks (RNNs) have good performance for speech synthesis when the text length is limited to a single sentence. The acoustic features obtained from the phoneme sequence converted from the text have good performance.

[0004] However, RNN-based speech synthesis models have poor feature analysis capabilities for phoneme sequences converted from complete text sequences, and are insufficient in capturing the acoustic features of details in the phoneme sequences, resulting in poor expression of the synthesized speech and inaccurate speech expression. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, and readable storage medium for acquiring acoustic features, which can achieve a grasp of the overall and detailed aspects of phoneme sequence information, improve the feature analysis capability of phoneme features, better meet the needs of emotional speech synthesis, and make the final synthesized speech more accurate and vivid.

[0006] In a first aspect, embodiments of the present invention provide a method for acquiring acoustic features, the method comprising:

[0007] Obtain the target phoneme sequence;

[0008] The target phoneme sequence is encoded using a split-channel self-attention mechanism to obtain a first feature matrix;

[0009] The first feature matrix is ​​decoded using a split-through self-attention mechanism to obtain the acoustic features of the target phoneme sequence.

[0010] Secondly, embodiments of the present invention provide an acoustic feature acquisition device, the device comprising:

[0011] The phoneme sequence acquisition module is used to acquire the target phoneme sequence;

[0012] The encoding module is used to perform encoding processing on the target phoneme sequence based on a split-flow self-attention mechanism to obtain a first feature matrix;

[0013] The decoding module is used to perform decoding processing on the first feature matrix based on a split-through self-attention mechanism to obtain the acoustic features of the target phoneme sequence.

[0014] Thirdly, embodiments of the present invention provide an electronic device, including: a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the method for acquiring acoustic features as described in the first aspect.

[0015] Fourthly, embodiments of the present invention provide a readable storage medium storing computer instructions, which, when executed by a processor, implement the method for acquiring acoustic features as described in the first aspect.

[0016] The embodiments of the present invention have the following advantages:

[0017] This invention provides a method for acquiring acoustic features. After acquiring a target phoneme sequence, the target phoneme sequence is encoded using a split-attention-based encoding process to obtain a first feature matrix. Then, the first feature matrix is ​​decoded using a split-attention-based decoding mechanism to obtain the acoustic features of the target phoneme sequence. The split-attention mechanism transforms the unified multi-head attention in multi-head self-attention mechanisms by performing scaling transformation on the key-value matrix and value matrix through preprocessing, thus converting it into split multi-head attention. This enables simultaneous modeling of information at different levels and scales, taking into account both the overall picture and the details, improving the feature analysis capability of phoneme features, and better adapting to the needs of emotional speech synthesis, resulting in more accurate and vivid expression of the final synthesized speech. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating an embodiment of the method for acquiring acoustic features according to the present invention is shown;

[0020] Figure 2 A comparison diagram of a multi-head self-attention mechanism and a shunting self-attention mechanism is shown;

[0021] Figure 3An architecture diagram of an embodiment of an acoustic model is shown;

[0022] Figure 4 A structural block diagram of an embodiment of an acoustic feature acquisition device according to the present invention is shown;

[0023] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Furthermore, it should be noted that the various data-related processes in the embodiments of this application are carried out in compliance with the relevant data protection laws and policies of the country where the location is located, and with the authorization granted by the owner of the corresponding device.

[0025] Speech synthesis technology is a technique that transforms computer-generated or externally input text information into fluent, understandable spoken Chinese. It is a cutting-edge technology in the field of Chinese information processing, primarily addressing the problem of converting text information into audible sound. Speech synthesis technology generally consists of three parts: a front-end, an acoustic model, and a vocoder. The acoustic model takes as input the phoneme sequence converted from text by the front-end and outputs the acoustic features of the speech. These acoustic features are then processed by the vocoder module to synthesize complete speech.

[0026] Current mainstream speech synthesis models are RNN-based and Transformer-based models. When the text length is limited to a single sentence, RNN-based speech synthesis models, which extract acoustic features from the phoneme sequence converted from the text, perform well in speech synthesis.

[0027] However, RNN-based speech synthesis models have poor feature analysis capabilities for phoneme sequences converted from complete text sequences, and Transformer-based speech synthesis models are not good at capturing detailed features of phoneme sequences, resulting in poor expression of the synthesized speech and inaccurate speech expression.

[0028] To solve the above problems, the present invention provides a method for obtaining acoustic features in the second stage of speech synthesis. By performing encoding-decoding processing based on split self-attention on the phoneme sequence transformed by the front end in sequence, the acoustic features of the phoneme sequence are obtained. It should be noted that in the specification, only matrices are used as variables to illustrate the method for obtaining the acoustic features, but it is not limited to variables in matrix form, and other forms such as vectors are not limited.

[0029] Referring to Figure 1 , a flowchart of an embodiment of a method for obtaining acoustic features according to the present invention is shown. The method may include:

[0030] Step 101, obtain a target phoneme sequence.

[0031] The phoneme refers to the smallest pronunciation unit in speech synthesis, and the phoneme sequence refers to a sequence composed of multiple phonemes. In an embodiment of the present invention, for the process of obtaining the second part of acoustic features in the speech synthesis process, that is, the acoustic features of the target phoneme sequence are obtained through an encoding-decoding manner according to the target phoneme sequence.

[0032] The target phoneme sequence is a sequence composed of phonemes transformed from natural language text through the first part of the front end of the speech synthesis process. For example, for the text "爱", the corresponding phoneme sequence is "ài".

[0033] Of course, the most straightforward phoneme sequence is not convenient for directly performing subsequent encoding-decoding processing. Therefore, the target phoneme sequence is usually subjected to embedding processing (embedding), including but not limited to word embedding and position embedding. The phoneme sequence after embedding processing can be a phoneme sequence in matrix form.

[0034] Step 102, perform encoding processing on the target phoneme sequence based on the split self-attention mechanism to obtain a first feature matrix.

[0035] The self-attention mechanism is a variant of the attention mechanism. It reduces the dependence on external information and is better at capturing the internal correlation of data or features. In applications such as text and characters, it mainly realizes long-distance dependence of context by calculating the mutual influence between characters. Specifically, it can obtain the embedding matrix of the input text, perform matrix transformation on the embedding matrix, obtain three matrices Q, K, and V, namely the query matrix (Q), the key-value matrix (K), and the value matrix (V), and calculate the mutual influence between characters based on Q, K, and V to achieve feature extraction or other technical problems based on encoding-decoding.

[0036] The split-attention mechanism is a novel variation of the self-attention mechanism. It obtains Q, K, and V by transforming the embedding matrix at different scales or dimensions. Unlike the Q, K, and V at the same scale in a single attention head in self-attention, the Q, K, and V in split-attention have different lengths in the attention head of a self-attention layer, which provides multi-scale token aggregation.

[0037] Reference Figure 2 The diagram shows a comparison between a multi-head self-attention mechanism and a split-head self-attention mechanism. In the multi-head self-attention mechanism on the left, the lengths of the input Q, K, and V are the same in the attention heads of a single self-attention layer, and all three are at the same scale. In contrast, the split-head self-attention mechanism on the right performs a downsampling scale transformation on the input matrix, resulting in Q, K, and V having different scales in the final input split-head self-attention model, providing multi-scale token aggregation for different attention heads. Therefore, the split-head self-attention mechanism can be considered a combination of scale transformation preprocessing and a self-attention mechanism.

[0038] Based on the above, the encoding process of the phoneme sequence based on splitting self-attention refers to obtaining a multi-scale phoneme matrix by performing a scale transformation on the target phoneme sequence obtained in step 101 after embedding, and then performing subsequent encoding processing on the multi-scale phoneme matrix based on the self-attention mechanism to obtain the first feature matrix of the target phoneme sequence.

[0039] In this embodiment of the invention, the phoneme sequence is encoded using a split-attention-based encoding process, while information at different levels and scales is modeled, taking into account both the overall picture and the details. The resulting acoustic features are more detailed and accurate, which is beneficial for more accurate and vivid sound synthesis in the subsequent third step of speech synthesis.

[0040] Step 103: Perform decoding processing on the first feature matrix based on the split-through self-attention mechanism to obtain the acoustic features of the target phoneme sequence.

[0041] The acoustic features refer to the physical quantities of the acoustic properties of speech extracted from speech according to a specific algorithm. They are also a general term for the acoustic performance of various sound elements, such as formant frequency, formant intensity and bandwidth, as well as the fundamental frequency and average speech power that represent the prosodic characteristics of speech. They are the output of the second step of speech synthesis.

[0042] Similar to the encoding described in step 102, the decoding process employed in this invention during the decoding stage is still based on a split-through self-attention mechanism. After scaling the first feature matrix, subsequent encoding processing based on the self-attention mechanism is performed to obtain the acoustic features of the target phoneme sequence.

[0043] Both the encoding and decoding stages employ a split-stream self-attention mechanism, transforming the input matrix at different scales to simultaneously model information at different levels. This approach balances overall and detailed understanding, enhances the ability to analyze phoneme features, and better adapts to the needs of emotional speech synthesis, resulting in more accurate final synthesized speech.

[0044] At the same time, downsampling the input matrix reduces its dimension, which reduces the amount of GPU memory used by the computer during encoding or decoding. This makes it easier for the encoding-decoding model to capture a larger amount of data at once during training, or for training long text speech synthesis, thus improving the efficiency of model training.

[0045] Optionally, step 102, which involves encoding the target phoneme sequence based on a split-channel self-attention mechanism to obtain a first feature matrix, includes:

[0046] Step S201: Encode and preprocess the target phoneme sequence at multiple scales to obtain the second feature matrix at multiple scales.

[0047] Step S202: Perform encoding processing on the second feature matrix based on a multi-head self-attention mechanism to obtain the first feature matrix.

[0048] Based on the split-stream self-attention mechanism described in step 102, it can be considered that the split-stream self-attention mechanism consists of an encoding preprocessing process and a self-attention-based encoding processing process. Therefore, the encoding preprocessing described in step S201 refers to performing the preprocessing process described in step 102 on the input phoneme sequence to obtain a second feature matrix at multiple scales.

[0049] In step S202, an extended form of the self-attention mechanism—the multi-head self-attention mechanism—is employed. Multi-head self-attention refers to a method that performs multiple attention operations on the same input to obtain multiple different outputs. In this embodiment of the invention, the phoneme sequence after embedding processing obtained in step 101 is processed by encoding preprocessing and then operated under multiple attention heads. This further expands the first feature matrix at multiple scales, thereby improving the ability to capture the acoustic features of the phoneme sequence.

[0050] Optionally, the encoding preprocessing in step S201 includes matrix transformation processing, downsampling processing, and feature enhancement processing; the third feature matrix includes a first query matrix, a first key-value matrix, and a first value matrix.

[0051] The encoding preprocessing of the target phoneme sequence at multiple scales in step S201 to obtain second feature matrices at multiple scales includes:

[0052] Step S2011: Perform matrix conversion processing on the second feature matrix to obtain a first query matrix;

[0053]

[0054] Among them, Q i represents the input of the i-th attention head, i.e., the first query matrix, during the process of step S202; P represents the phoneme sequence obtained in step 101; represents the preset conversion parameter matrix from matrix P to Q i ; 1 < i < N + 1, i is an integer, and N is the number of attention heads in the encoding process.

[0055] Step S2012: Perform downsampling processing and matrix conversion processing on the second feature matrix according to the following formula to obtain a first key-value matrix:

[0056]

[0057] Among them, K i represents the input of the i-th attention head, i.e., the first key-value matrix, during the encoding processing based on multi-head self-attention; DownSampling() represents the downsampling processing process; the parameter r i represents the hyperparameter of the downsampling processing in the i-th attention head during the encoding processing based on multi-head self-attention; represents the preset conversion parameter matrix from matrix P to K i .

[0058] Step S2013: Perform matrix conversion processing, downsampling processing and feature enhancement processing on the second feature matrix according to the following formula to obtain a first value matrix:

[0059]

[0060] Among them, V i represents the input of the i-th attention head, i.e., the first value matrix, during the encoding processing based on multi-head self-attention; V i' represents the intermediate matrix in the process from matrix P to matrix V i ; represents the preset conversion parameter matrix from matrix P to V i' ; FE() represents the feature enhancement processing process.

[0061] Steps S2011-S2013 above describe the preprocessing of the input matrix for each attention head during the multi-head attention-based encoding process. For matrix K... i It is formed by first downsampling the embedded phoneme sequence P and then combining it with a pre-set transformation parameter matrix. The downsampling process, consisting of convolution and normalization, downsamples matrix P along its time dimension. The output time dimension is controlled by the parameter r. For example, if r = 2, the time dimension of P will be halved. The larger r is, the more the time dimension is compressed. When r = 1, matrix P remains unchanged. By assigning different r values ​​to each attention head in the module, matrices K of different scales can be obtained, thus acquiring feature information at different scales. Similarly, to obtain matrix Vi, the same downsampling process is performed, followed by multiplication with matrix P. Multiplication is performed. Building upon this, to further enhance the encoding capability of local information, a feature enhancement module FE is added after matrix V to further extract information. The FE module has the same basic structure as the DownSampling module, the difference being that it replaces the one-dimensional convolution with a one-dimensional depth-wise convolution. Correspondingly, matrix Q... i It is then obtained simply by multiplying matrix P and used as a reference.

[0062] The above-mentioned encoding preprocessing process preprocesses the embedded phoneme sequence at multiple scales, which not only enhances the encoding ability of local information during the encoding process, but also takes into account the overall and local features of the phoneme sequence, making the final acoustic features more accurate and richer.

[0063] Optionally, step 103, which involves performing decoding processing on the first feature matrix based on a split-through self-attention mechanism to obtain the acoustic features of the target phoneme sequence, includes:

[0064] Step S301: Decode and preprocess the first feature matrix at multiple scales to obtain a third feature matrix at multiple scales.

[0065] Step S302: Perform multi-head self-attention-based decoding on the third feature matrix to obtain the acoustic features.

[0066] The decoding process based on split self-attention consists of decoding preprocessing and decoding processing based on multi-head self-attention. By preprocessing the first feature matrix obtained from the encoding process, a third feature matrix at multiple scales is obtained. This step further enhances the grasp of the overall and local information of the phoneme sequence during the acquisition of acoustic features.

[0067] Optionally, the decoding preprocessing described in step S301 includes matrix conversion processing, downsampling processing, and feature enhancement processing; the third feature matrix includes a second query matrix, a second key matrix, and a second value matrix;

[0068] Performing the decoding preprocessing on the first feature matrix at multiple scales to obtain third feature matrices at multiple scales includes:

[0069] Performing matrix conversion processing on the first feature matrix to obtain a second query matrix;

[0070]

[0071] where Q j represents the input of the j-th attention head, i.e., the second query matrix, during the decoding processing based on multi-head attention; O represents the first feature matrix; represents the preset conversion parameter matrix from matrix O to Q j ; 1 < j < N + 1, i is an integer, and N is the number of heads of the shunted self-attention mechanism during the decoding processing.

[0072] Performing downsampling processing and matrix conversion processing on the first feature matrix according to the following formula to obtain a second key matrix:

[0073]

[0074] where K j represents the input of the j-th attention head, i.e., the second key matrix, during the decoding processing based on multi-head self-attention; DownSampling represents the downsampling processing process; the parameter r j represents the hyperparameter of the downsampling processing in the j-th attention head during the decoding processing based on multi-head self-attention; represents the preset conversion parameter matrix from matrix O to K j ;

[0075] Performing matrix conversion processing, downsampling processing, and feature enhancement processing on the first feature matrix according to the following formula to obtain a second value matrix:

[0076]

[0077] where V j represents the input of the j-th attention head, i.e., the second value matrix, during the decoding processing based on multi-head self-attention; V j' represents the intermediate matrix during the process from matrix O to matrix V j ; represents the process from matrix O to V j'The pre-defined transformation parameter matrix; FE represents the feature enhancement process.

[0078] Optionally, step 103, which involves performing decoding processing on the first feature matrix based on a split-through self-attention mechanism to obtain the acoustic features of the target phoneme sequence, may include:

[0079] Step 1030: Based on the first feature matrix, predict the duration information of the target phoneme matrix to generate a fourth feature matrix;

[0080] Step 10300: Perform decoding processing based on split self-attention on the fourth feature matrix to obtain the acoustic features of the phoneme sequence.

[0081] In the process of acquiring acoustic features, the duration of each phoneme in the phoneme sequence is predicted or duration information is provided by external tools. The duration information of the phoneme sequence is added to the first feature matrix obtained by encoding processing, and the first feature matrix is ​​expanded into a fourth feature matrix, that is, the fourth feature matrix contains the duration information of the phoneme sequence.

[0082] Optionally, step 102, which involves sequentially performing multi-layer self-attention-based encoding on the first feature matrix to obtain the second feature matrix of the phoneme sequence, may further include:

[0083] The first feature matrix is ​​subjected to a multi-layer self-attention-based encoding process in sequence to obtain the second feature matrix of the phoneme sequence.

[0084] Step 103, which involves decoding the first feature matrix using a split-channel self-attention mechanism to obtain the acoustic features of the target phoneme sequence, may further include:

[0085] The first feature matrix is ​​serially decoded using a split-through self-attention mechanism to obtain the acoustic features of the target phoneme sequence.

[0086] The encoding process based on split-stream self-attention can be performed serially in multiple layers. Similarly, the decoding process based on split-stream self-attention can also be performed serially in multiple layers, as shown in Table 2 for example, the hyperparameter settings in the encoding-decoding process.

[0087] Table 2

[0088] Encoder_SSA_1 Encoder_SSA_2 Encoder_SSA_3 Decoder_SSA_1_4 head 2 2 3 4 r 1,2 2,4 1,2,4 1,2,4,8

[0089] In the example shown in Table 2, Encoder_SSA represents encoding processing based on split-stream self-attention, and Decoder_SSA represents decoding processing based on split-stream self-attention. The table includes three encoding processing layers and four decoding processing layers, each of which is set with the number of attention heads (head) and the downsampling time parameter (r).

[0090] Reference Figure 3 An architectural diagram of an embodiment of an acoustic model is shown, the acoustic model being used to... Figure 1 The method for acquiring acoustic features describes the acquisition of acoustic features. The model consists of an embedding layer, a coding layer based on split-through self-attention, a position embedding layer, and a decoding layer based on split-through self-attention.

[0091] The single-head attention architecture of the encoding and decoding layers is as follows: Figure 3 As shown on the right, FE represents feature enhancement processing, concat represents merging processing, and linear represents linear processing.

[0092] The embedding layer is used to embed the input phoneme sequence and output the embedded phoneme sequence. The embedding process includes, but is not limited to, character embedding and position embedding.

[0093] The split-attention-based coding layer receives the output of the embedding layer. The number of attention heads can be single or multiple. First, the coding preprocessing described in steps S2011-S2013 is executed to obtain the first query matrix (Q). i ), first key-value matrix (K) i ), first value matrix (V) i ), to the Q i K i V i After linear processing, it is input into the attention layer, then sequentially into the merging layer and the linear layer, and finally outputs the first feature matrix of the phoneme sequence as described in step 102.

[0094] Similar to the embedding layer, the position embedding layer is used to receive the output of the coding layer and perform position embedding processing.

[0095] The split-attention-based decoding layer receives the output of the position embedding layer. The number of attention heads can be single or multiple. First, the decoding preprocessing described in steps S3011-S3013 is executed to obtain the first query matrix (Q). j ), first key-value matrix (K) j ), first value matrix (V) j ), and the matrix Q j K j V jAfter linear processing, it is input into the attention layer, then sequentially into the merging layer and the linear layer, and finally outputs the acoustic features of the phoneme sequence as described in step 103.

[0096] In summary, the embodiments of the present invention provide a method for acquiring acoustic features. After acquiring the target phoneme sequence, the target phoneme sequence is encoded using a split-attention-based encoding process to obtain a first feature matrix. Then, the first feature matrix is ​​decoded using a split-attention-based decoding mechanism to obtain the acoustic features of the target phoneme sequence. The split-attention mechanism transforms the unified multi-head attention in the multi-head self-attention mechanism into split multi-head attention by performing scaling transformation on the key-value matrix and value matrix through preprocessing. This enables simultaneous modeling of information at different levels and scales, taking into account both the overall picture and the details, improving the feature analysis capability of phoneme features, and better adapting to the needs of emotional speech synthesis, resulting in more accurate and vivid expression of the final synthesized speech.

[0097] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0098] Reference Figure 4 The diagram illustrates a structural block diagram of an embodiment of an acoustic feature acquisition device according to the present invention. The device 300 may include:

[0099] Phoneme sequence acquisition module 301 is used to acquire target phoneme sequences;

[0100] The encoding module 302 is used to perform encoding processing on the target phoneme sequence based on a split-stream self-attention mechanism to obtain a first feature matrix;

[0101] The decoding module 303 is used to perform decoding processing on the first feature matrix based on the split self-attention mechanism to obtain the acoustic features of the target phoneme sequence.

[0102] Optionally, the encoding module may include:

[0103] The encoding preprocessing module is used to perform encoding preprocessing on the target phoneme sequence at multiple scales to obtain a second feature matrix at multiple scales;

[0104] The multi-head self-attention-based encoding module is used to encode the second feature matrix using a multi-head self-attention mechanism to obtain the first feature matrix.

[0105] Optionally, the encoding preprocessing in the encoding preprocessing module includes matrix transformation processing, downsampling processing, and feature enhancement processing; the second feature matrix includes a first query matrix, a first key-value matrix, and a first value matrix;

[0106] The encoding preprocessing module may include:

[0107] The first query matrix generation module is used to perform downsampling and matrix transformation on the second feature matrix according to the following formula to obtain the first key-value matrix:

[0108]

[0109] Among them, K i In the multi-head self-attention encoding process, the input of the i-th attention head is the first key-value matrix; DownSampling represents the downsampling process; parameter r i This represents the hyperparameters of downsampling in the i-th attention head during the multi-head self-attention-based encoding process; Represents the process from matrix P to K i The pre-defined transformation parameter matrix;

[0110] The first key-value matrix generation module is used to perform downsampling and matrix transformation on the second feature matrix according to the following formula to obtain the first key-value matrix:

[0111]

[0112] Among them, K i In the multi-head self-attention encoding process, the input of the i-th attention head is the first key-value matrix; DownSampling represents the downsampling process; parameter r i This represents the hyperparameters of downsampling in the i-th attention head during the multi-head self-attention-based encoding process; Represents the process from matrix P to K i The pre-defined transformation parameter matrix;

[0113] The first-value matrix generation module is used to perform matrix transformation, downsampling, and feature enhancement on the second feature matrix according to the following formula to obtain the first-value matrix:

[0114]

[0115] Among them, Vi denotes the input of the i-th attention head, i.e., the first value matrix, during the encoding process based on multi-head self-attention; V i' denotes the intermediate matrix during the process from matrix P to matrix V i ; denotes the process from matrix P to V i' ; the preset transformation parameter matrix from matrix P to V; FE denotes the feature enhancement process.

[0116] Optionally, the decoding module may include:

[0117] a decoding preprocessing module for performing decoding preprocessing on the first feature matrix at multiple scales to obtain third feature matrices at multiple scales;

[0118] a decoding module based on multi-head self-attention for performing decoding processing based on multi-head self-attention on the third feature matrix to obtain the acoustic features.

[0119] Optionally, the decoding preprocessing in the decoding preprocessing module includes matrix transformation processing, downsampling processing, and feature enhancement processing; the third feature matrix includes a second query matrix, a second key matrix, and a second value matrix;

[0120] The decoding preprocessing module may include:

[0121] a second query matrix generation module for performing matrix transformation processing on the first feature matrix to obtain a second query matrix;

[0122]

[0123] where Q j denotes the input of the j-th attention head, i.e., the second query matrix, during the decoding process based on multi-head attention; O denotes the first feature matrix; denotes the preset transformation parameter matrix from matrix O to Q j ; 1 < j < N + 1, i is an integer, and N is the number of attention heads of the shunt self-attention mechanism in the decoding process;

[0124] a second key matrix generation module for performing downsampling processing and matrix transformation processing on the first feature matrix according to the following formula to obtain a second key matrix:

[0125]

[0126] where K jThis indicates that the input of the j-th attention head, i.e., the second key-value matrix, is used in the multi-head self-attention-based decoding process; DownSampling represents the downsampling process; parameter r j This represents the hyperparameters of downsampling in the j-th attention head during the multi-head self-attention-based decoding process. Represents the transition from matrix O to K j The pre-defined transformation parameter matrix;

[0127] The second value matrix generation module is used to perform matrix transformation, downsampling, and feature enhancement on the first feature matrix according to the following formula to obtain the second value matrix:

[0128]

[0129] Among them, V j This indicates that during the multi-head self-attention-based decoding process, the input of the j-th attention head is the second value matrix; V j' Represents the transformation from matrix O to matrix V j The intermediate matrix in the processing; Represents the transition from matrix O to V j' The pre-defined transformation parameter matrix; FE represents the feature enhancement process.

[0130] Optionally, the device may further include:

[0131] The duration prediction module is used to predict the duration information of the target phoneme matrix based on the first feature matrix and generate a fourth feature matrix.

[0132] Correspondingly, the decoding module includes:

[0133] The decoding module that integrates duration information is used to perform decoding processing based on split self-attention on the fourth feature matrix to obtain the acoustic features of the phoneme sequence.

[0134] As the apparatus embodiment is basically similar to the method embodiment, it is described in a relatively simple manner. For relevant details, please refer to the description of the method embodiment.

[0135] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0136] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0137] Reference Figure 5 The electronic device 600 may include one or more of the following components: a processing component 602, a memory 604, a power supply component 606, a multimedia component 608, an audio component 610, an input / output (I / O) interface 612, a sensor component 614, and a communication component 616.

[0138] Processing component 602 typically controls the overall operation of electronic device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0139] Memory 604 is used to store various types of data to support the operation of electronic device 600. Examples of such data include instructions for any application or method operating on electronic device 600, contact data, phonebook data, messages, pictures, multimedia, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0140] Power supply component 606 provides power to various components of electronic device 600. Power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 600.

[0141] Multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a multimedia mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0142] Audio component 610 is used to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) used to receive external audio signals when electronic device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0143] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0144] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of electronic device 600. For example, sensor assembly 614 can detect the on / off state of electronic device 600, the relative positioning of components such as the display and keypad of electronic device 600, changes in position of electronic device 600 or a component of electronic device 600, the presence or absence of user contact with electronic device 600, orientation or acceleration / deceleration of electronic device 600, and temperature changes of electronic device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0145] Communication component 616 facilitates wired or wireless communication between electronic device 600 and other devices. Electronic device 600 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0146] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement a method for acquiring acoustic features provided in this disclosure.

[0147] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by a processor 620 of an electronic device 600 to perform the above-described method. For example, the non-transitory storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0148] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the method for acquiring acoustic features.

[0149] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this disclosure may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0150] Similarly, it should be understood that, in order to simplify this disclosure and aid in understanding one or more of the various inventive aspects, in the foregoing description of exemplary embodiments of the present disclosure, various features of the embodiments of the present disclosure are sometimes grouped together in a single embodiment, figure, or description thereof. However, this approach to disclosure should not be construed as reflecting an intention that the claimed embodiments of the present disclosure require more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the present disclosure.

[0151] Those skilled in the art will understand that the modules in the device of the embodiments can be adaptively modified and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be employed to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0152] The various component embodiments of this disclosure can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the sorting device according to the embodiments of this disclosure. Embodiments of this disclosure can also be implemented as device or apparatus programs for performing some or all of the methods described herein. Such programs implementing embodiments of this disclosure can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0153] It should be noted that the above embodiments are illustrative of embodiments of this disclosure and not restrictive of embodiments of this disclosure, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Embodiments of this disclosure can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0154] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0155] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the embodiments of the present disclosure. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the embodiments of the present disclosure should be included within the protection scope of the embodiments of the present disclosure.

[0156] The above description is merely a specific implementation of the embodiments of this disclosure, but the protection scope of the embodiments of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this disclosure should be included within the protection scope of the embodiments of this disclosure. Therefore, the protection scope of the embodiments of this disclosure should be determined by the protection scope of the claims.

Claims

1. A method for acquiring acoustic features, characterized in that, include: Obtain the target phoneme sequence; The target phoneme sequence is encoded using a split-self-attention encoding process to obtain a first feature matrix; The first feature matrix is ​​decoded using a split-through self-attention mechanism to obtain the acoustic features of the target phoneme sequence. The encoding process based on a split-channel self-attention mechanism on the target phoneme sequence to obtain the first feature matrix includes: The target phoneme sequence is preprocessed by encoding at multiple scales to obtain a second feature matrix at multiple scales; The second feature matrix is ​​encoded using a multi-head self-attention mechanism to obtain the first feature matrix.

2. The method according to claim 1, characterized in that, The encoding preprocessing includes matrix transformation processing, downsampling processing, and feature enhancement processing; the second feature matrix includes a first query matrix, a first key-value matrix, and a first value matrix; The step of encoding and preprocessing the target phoneme sequence at multiple scales to obtain a second feature matrix at multiple scales includes: Perform matrix transformation on the second feature matrix to obtain the first query matrix; Q i =PW i Q Among them, Q i represents the input of the i-th attention head, that is, the first query matrix, during the encoding process based on multi-head attention; P represents the target phoneme sequence; W i Q represents a preset transformation parameter matrix from matrix P to Q i ; 1 < i < N + 1, where i is an integer and N is the number of attention heads in the shunted self-attention mechanism during the encoding process; The second feature matrix is ​​downsampled and transformed according to the following formula to obtain the first key-value matrix: K i =DownSampling(P,r i )W i K Among them, K i In the multi-head self-attention encoding process, the input of the i-th attention head is the first key-value matrix; DownSampling represents the downsampling process; parameter r i W represents the hyperparameters of downsampling in the i-th attention head during multi-head self-attention-based encoding; i K Represents the process from matrix P to K i The pre-defined transformation parameter matrix; The second feature matrix is ​​subjected to matrix transformation, downsampling, and feature enhancement processing according to the following formula to obtain the first value matrix: V i' =DownSampling(P,r i )W i V ,V i =V i' +FE(V i' ) Among them, V i This indicates that during the multi-head self-attention-based encoding process, the input of the i-th attention head is the first value matrix; V i' Represents the transformation from matrix P to matrix V i Intermediate matrices in the processing; W i V Represents the transition from matrix P to V i' The pre-defined transformation parameter matrix; FE represents the feature enhancement process.

3. The method according to claim 1, characterized in that, The step of performing decoding processing on the first feature matrix based on a split-channel self-attention mechanism to obtain the acoustic features of the target phoneme sequence includes: The first feature matrix is ​​decoded and preprocessed at multiple scales to obtain a third feature matrix at multiple scales; The third feature matrix is ​​subjected to multi-head self-attention-based decoding to obtain acoustic features.

4. The method according to claim 3, characterized in that, The decoding preprocessing includes matrix transformation processing, downsampling processing, and feature enhancement processing; the third feature matrix includes a second query matrix, a second key-value matrix, and a second value matrix; The step of decoding and preprocessing the first feature matrix at multiple scales to obtain a third feature matrix at multiple scales includes: Perform matrix transformation on the first feature matrix to obtain the second query matrix; Among them, Q j represents the input of the j-th attention head, i.e., the second query matrix, in the decoding process based on the multi-head attention; O represents the first feature matrix; represents the preset transformation parameter matrix from the matrix O to Q j ; 1 < j < N + 1, i is an integer, and N is the number of heads of the attention of the shunted self-attention mechanism in the decoding process; The first feature matrix is ​​downsampled and transformed using the following formula to obtain the second key-value matrix: Among them, K j This indicates that the input of the j-th attention head, i.e., the second key-value matrix, is used in the multi-head self-attention-based decoding process; DownSampling represents the downsampling process; parameter r j This represents the hyperparameters of downsampling in the j-th attention head during the multi-head self-attention-based decoding process. Represents the transition from matrix O to K j The pre-defined transformation parameter matrix; The first feature matrix is ​​subjected to matrix transformation, downsampling, and feature enhancement processing according to the following formula to obtain the second value matrix: Among them, V j This indicates that during the multi-head self-attention-based decoding process, the input of the j-th attention head is the second value matrix; V j' Represents the transformation from matrix O to matrix V j The intermediate matrix in the processing; Represents the transition from matrix O to V j' The pre-defined transformation parameter matrix; FE represents the feature enhancement process.

5. The method according to claim 1, characterized in that, The step of performing decoding processing on the first feature matrix based on a split-channel self-attention mechanism to obtain the acoustic features of the target phoneme sequence includes: Based on the first feature matrix, the duration information of the target phoneme matrix is ​​predicted to generate a fourth feature matrix; The fourth feature matrix is ​​subjected to a decoding process based on split-through self-attention to obtain the acoustic features of the phoneme sequence.

6. The method according to claim 1, characterized in that, The step of encoding the target phoneme sequence based on a split-channel self-attention mechanism to obtain the first feature matrix further includes: The target phoneme sequence is serially encoded using a split-channel self-attention mechanism to obtain the first feature matrix. The step of performing decoding processing on the first feature matrix based on a split-channel self-attention mechanism to obtain the acoustic features of the target phoneme sequence includes: The first feature matrix is ​​serially decoded using a split-through self-attention mechanism to obtain the acoustic features of the target phoneme sequence.

7. A device for acquiring acoustic features, characterized in that, include: The phoneme sequence acquisition module is used to acquire the target phoneme sequence; The encoding module is used to perform encoding processing on the target phoneme sequence based on a split-flow self-attention mechanism to obtain a first feature matrix; The decoding module is used to perform decoding processing on the first feature matrix based on the split self-attention mechanism to obtain the acoustic features of the target phoneme sequence; The encoding module includes: The encoding preprocessing module is used to perform encoding preprocessing on the target phoneme sequence at multiple scales to obtain a second feature matrix at multiple scales; The multi-head self-attention-based encoding module is used to encode the second feature matrix using a multi-head self-attention mechanism to obtain the first feature matrix.

8. An electronic device, characterized in that, include: A processor and a memory, wherein the processor executes a computer program stored in the memory to implement the method of any one of claims 1 to 6.

9. A readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the device, the device is able to perform the method as described in any one of claims 1 to 6.

Images