Pronunciation detection method and related apparatus, electronic device, and storage medium

Abstract

This application discloses a pronunciation detection method and related apparatus, electronic device, and storage medium. The pronunciation detection method includes: acquiring test speech collected from a test subject reading target text, and acquiring a phoneme sequence obtained by phoneme conversion of the target text; wherein the test speech contains several audio frames; extracting phoneme features of each phoneme in the phoneme sequence to obtain a first phoneme feature sequence, and determining multiple downsampling rates; for each downsampling rate, encoding several audio frames based on the downsampling rate to obtain a prosodic feature sequence corresponding to the downsampling rate; for the prosodic feature sequences corresponding to each downsampling rate, fusing the first phoneme feature sequence and the prosodic feature sequence to obtain a second phoneme feature sequence; and predicting based on each second phoneme feature sequence to obtain a pronunciation score for the test speech. The above scheme can improve the accuracy of pronunciation detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a pronunciation detection method and related device, electronic device and storage medium. BACKGROUND

[0002] To adapt to the trend of globalization, self-study foreign language systems are becoming more and more attractive. Computer-Assisted Pronunciation Training (CAPT) has become an effective tool for non-native speakers to learn different foreign languages.

[0003] In this process, pronunciation detection needs to be performed on the test speech of the testee reading a specific text to evaluate the pronunciation level of the testee. However, the current pronunciation detection method is mostly rough and still difficult to accurately evaluate the pronunciation level. Therefore, how to improve the accuracy of pronunciation detection becomes a problem to be solved. SUMMARY

[0004] The technical problem solved by the present application is to provide a pronunciation detection method and related device, electronic device and storage medium, which can improve the accuracy of pronunciation detection.

[0005] To solve the above technical problem, the first aspect of the present application provides a pronunciation detection method, comprising: acquiring test speech collected by a testee reading a target text, and acquiring a phoneme sequence obtained by performing phoneme conversion on the target text; wherein the test speech comprises a plurality of audio frames; extracting phoneme features of each phoneme in the phoneme sequence to obtain a first phoneme feature sequence, and determining a plurality of down-sampling rates; for each down-sampling rate, encoding the plurality of audio frames based on the down-sampling rate to obtain a prosody feature sequence corresponding to the down-sampling rate; for the prosody feature sequences corresponding to each down-sampling rate respectively, fusing the first phoneme feature sequence and the prosody feature sequence to obtain a second phoneme feature sequence; and predicting based on each second phoneme feature sequence to obtain a pronunciation score of the test speech.

[0006] To solve the above technical problems, the first aspect of the present application provides two pronunciation detection devices, comprising: an acquisition module, an extraction module, a determination module, an encoding module, a fusion module and a prediction module, the acquisition module is used to acquire test speech collected by a subject reading a target text, and acquire a phoneme sequence obtained by phoneme conversion of the target text; wherein the test speech contains a plurality of audio frames; the extraction module is used to extract phoneme features of each phoneme in the phoneme sequence to obtain a first phoneme feature sequence; the determination module is used to determine a plurality of down-sampling rates; the encoding module is used to encode the plurality of audio frames based on the down-sampling rate for each down-sampling rate to obtain a prosody feature sequence corresponding to the down-sampling rate; the fusion module is used to fuse the first phoneme feature sequence and the prosody feature sequence for the prosody feature sequence corresponding to each down-sampling rate to obtain a second phoneme feature sequence; and the prediction module is used to predict based on each second phoneme feature sequence to obtain a pronunciation score of the test speech.

[0007] To solve the above technical problems, the third aspect of the present application provides an electronic device, comprising a memory and a processor coupled to each other, the memory stores program instructions, and the processor is used to execute the program instructions to realize the pronunciation detection method of the first aspect.

[0008] To solve the above technical problems, the fourth aspect of the present application provides a computer-readable storage medium, which stores program instructions capable of being executed by a processor, and the program instructions are used to realize the pronunciation detection method of the first aspect.

[0009] The above scheme extracts test speech collected by a subject reading a target text, and acquires a phoneme sequence obtained by phoneme conversion of the target text, and the test speech includes a plurality of audio frames, then extracts phoneme features of each phoneme in the phoneme sequence to obtain a first phoneme feature sequence, and determines a plurality of down-sampling rates, so as to encode the plurality of audio frames based on the down-sampling rate for each down-sampling rate to obtain a prosody feature sequence corresponding to the down-sampling rate, and then fuse the first phoneme feature sequence and the prosody feature sequence for the prosody feature sequence corresponding to each down-sampling rate to obtain a second phoneme feature sequence, and predict based on each second phoneme feature sequence to obtain a pronunciation score of the test speech. Since the prosody feature sequences corresponding to various down-sampling rates are obtained by encoding with different down-sampling rates, the pronunciation detection can be assisted by multiple levels of prosody features, which helps to improve the refinement degree of pronunciation detection. Therefore, the accuracy of pronunciation detection can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a flowchart of an embodiment of the pronunciation detection method of the present application;

[0011] Figure 2is a schematic diagram of a framework of an embodiment of a prosody encoder;

[0012] Figure 3 is a schematic diagram of a process of an embodiment of a pronunciation detection method of the present application;

[0013] Figure 4 is a schematic diagram of a process of another embodiment of a pronunciation detection method of the present application;

[0014] Figure 5 is a schematic diagram of a framework of an embodiment of a pronunciation detection apparatus of the present application;

[0015] Figure 6 is a schematic diagram of a framework of an embodiment of an electronic device of the present application;

[0016] Figure 7 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0017] The schemes of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0018] In the following description, specific details are set forth in order to provide a thorough understanding of the present application. However, persons having ordinary skill in the art will appreciate that the present application can be practiced without the specific details.

[0019] The terms "system" and "network" are often used interchangeably herein. The term "and / or" herein is merely used to represent an associated relationship between associated objects, and means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " herein generally represents an "or" relationship between the associated objects. In addition, "multiple" herein means two or more than two.

[0020] Please refer to Figure 1 , Figure 1 is a schematic diagram of a process of an embodiment of a pronunciation detection method of the present application.

[0021] Specifically, it can include the following steps:

[0022] Step S11: obtaining the test speech collected by the subject reading the target text, and obtaining the phoneme sequence obtained by phoneme conversion of the target text.

[0023] In one implementation scenario, the native language of the subject can or can not be the language to which the target text belongs, which is set according to the actual application scenario, and is not limited herein. For example, when applied to an autonomous foreign language learning system, the native language of the subject can not be the language to which the target text belongs, such as the native language of the subject being Chinese and the language to which the target text belongs being English. Or, when applied to a native language test (such as a Chinese level test), the native language of the subject can also be the language to which the target text belongs, such as the native language of the subject being Chinese and the language to which the target text belongs being Chinese. Other cases can be similarly deduced, and will not be repeated herein.

[0024] In one implementation scenario, when the subject reads the target text, the subject's voice can be collected to obtain the test voice. In addition, in order to improve the audio quality of the test voice and further improve the accuracy of subsequent pronunciation detection, the subject can be collected in a space that isolates environmental noise, or a high-quality sound pickup device such as a noise reduction microphone can be used for voice collection, which is not limited herein.

[0025] In the embodiments of the present disclosure, the test voice can include a plurality of audio frames. It should be noted that in actual application, the number of audio frames contained in the test voice is different according to different recording lengths, and the specific number is not limited herein. In addition, the original frame rate of the test voice is not limited to a specific frame rate in actual application, such as 10 frames per second, or 20 frames per second, 30 frames per second, etc., which is not limited herein.

[0026] In one implementation scenario, a text-to-phoneme conversion tool such as phonemizer or Text-to-Phoneme can be used to convert the target text into a phoneme sequence.

[0027] In another implementation scenario, different from the foregoing implementation, in order to further improve the accuracy of phoneme conversion, the target text can also be aligned with the test voice to obtain a phoneme sequence. For example, the target text can be aligned with the test voice by an HMM-DNN model to obtain a pronunciation sequence corresponding to the target text, i.e., a phoneme sequence. The above method of aligning the target text with the test voice to obtain a phoneme sequence can alleviate the influence of one word with multiple pronunciations on phoneme conversion as much as possible, which helps to improve the accuracy of phoneme conversion.

[0028] Step S12: Extracting phoneme features of each phoneme in the phoneme sequence to obtain a first phoneme feature sequence, and determining a plurality of down-sampling rates.

[0029] It should be noted that the first phoneme feature sequence includes the phoneme features of each phoneme contained in the phoneme sequence. In addition, the phoneme features specifically include feature information related to the pronunciation of the phoneme.

[0030] In one implementation scenario, the embedding representation of each phoneme in the phoneme sequence can be obtained first, and then the embedding representation of each phoneme can be processed based on the self-attention mechanism considering the correlation between the phonemes to obtain the phoneme features of each phoneme. It should be noted that the specific process of processing the embedding representation of each phoneme can refer to the technical details of the self-attention mechanism, which will not be repeated here.

[0031] In another implementation scenario, different from the foregoing manner, in order to further improve the accuracy of the phoneme features, after obtaining the embedding representation of each phoneme, the position representation of each phoneme can be obtained, and then the embedding representation and the position representation of the phoneme are fused (such as added) to obtain the hidden layer representation of the phoneme. On this basis, the hidden layer representation of each phoneme is processed based on the self-attention mechanism to obtain the phoneme features of each phoneme.

[0032] It should be noted that the plurality of down-sampling rates can each be greater than the original frame rate of the test speech, and the plurality of down-sampling rates are each different.

[0033] In one implementation scenario, several down-sampling rates greater than the original frame rate can be randomly selected. For example, in the case where the original frame rate is 10 frames per second, 20 frames per second, 40 frames per second, 60 frames per second, 80 frames per second, etc. can be randomly selected. Of course, the above examples are only several possible selection methods in actual application, and do not limit the actual selection method of the down-sampling rate.

[0034] In another implementation scenario, in order to more accurately model the multi-level rhythm, a plurality of multiples can be used to multiply the original frame rate to obtain a plurality of down-sampling rates. The plurality of multiples can include but are not limited to 8 times, 32 times, 64 times, etc., and the setting method of the multiple is not limited here. It should be noted that the specific setting of the plurality of multiples can consider the relationship between the duration of various levels such as phonemes, words, sentences and the original frame rate. For example, the larger the original frame rate, the smaller the interval duration between adjacent frames, and accordingly, the larger the number of frames required to model the same duration of phoneme level (or word level, or sentence level), so the multiple can be set to be appropriately larger. Conversely, the smaller the original frame rate, the larger the interval duration between adjacent frames, and accordingly, the smaller the number of frames required to model the same duration of phoneme level (or word level, or sentence level), so the multiple can be set to be appropriately smaller. Other cases can be similarly extended, which will not be repeated here.

[0035] Step S13: encode the several audio frames based on the down-sampling rate to obtain the prosody feature sequence corresponding to the down-sampling rate.

[0036] It should be noted that the prosody feature sequence can include at least one prosody feature, and the prosody feature includes feature information related to pronunciation prosody. In addition, the greater the down-sampling rate, the greater the level granularity represented by the prosody feature. For example, still taking the setting of 8 times, 32 times and 64 times as an example, the prosody feature corresponding to the down-sampling rate of 8 times can be regarded as a prosody feature at the phoneme level, that is, it includes feature information related to prosody at the phoneme level. The prosody feature corresponding to the down-sampling rate of 32 times can be regarded as a prosody feature at the word level, that is, it includes feature information related to prosody at the word level. The prosody feature corresponding to the down-sampling rate of 64 times can be regarded as a prosody feature at the sentence level, that is, it includes feature information related to prosody at the sentence level. In addition, in the embodiment of the present disclosure, "several audio frames" in "encoding the several audio frames based on the down-sampling rate to obtain the prosody feature sequence corresponding to the down-sampling rate" refers to the audio frames contained in the test speech.

[0037] Specifically, for various down-sampling rates, the prosody encoder corresponding to the down-sampling rate can be used to encode the several audio frames to obtain the prosody feature sequence corresponding to the down-sampling rate.

[0038] In one implementation scenario, the prosody encoder can include but is not limited to an encoding network such as a forward encoding network, a reverse encoding network, and a feature extraction network, which are not limited here. Please refer to Figure 2 Figure 2 is a schematic diagram of the framework of an embodiment of the prosody encoder. As Figure 2 shown, the function of the forward encoding network is to perform complex nonlinear transformation on the original input audio frame, which can specifically include convolutional layers and pooling layers. It should be noted that the convolutional layer can use causal convolution, which aims to obtain the upper field of view without looking at the lower field of view. The pooling layer is to down-sample the input features using the down-sampling rate corresponding to the prosody encoder to obtain the prosody features. In addition, the network structure of the reverse encoding network is the same as that of the forward encoding network, and also includes convolutional layers and pooling layers. The difference is that the convolutional layer in the reverse encoding network can only see the lower field of view and cannot see the upper field of view. The feature extraction network can be specifically constructed as a one-way recurrent neural network (RNN) for further time coding on the output features of the forward encoding network frame by frame. The working principle of the prosody encoder can refer to the technical details of the CPC (Contrastive Predictive Coding) model, which will not be described here.

[0039] ​In one implementation scenario, as mentioned earlier, prosodic feature sequences corresponding to different downsampling rates are obtained by encoding several audio frames using a prosodic encoder at the corresponding downsampling rate. Furthermore, the prosodic encoder is trained using contrastive predictive coding based on a second sample of speech, which includes several second sample audio frames. It should be noted that the specific details of contrastive predictive coding can be found in its technical specifications and will not be elaborated upon here. This method, where prosodic feature sequences corresponding to different downsampling rates are encoded by a prosodic encoder at the corresponding downsampling rate, and the prosodic encoder is trained using contrastive predictive coding based on a second sample of speech (which includes several second sample audio frames), allows for unsupervised training of the prosodic encoder. This eliminates the need for prosodic-annotated sample speech data, significantly alleviating the training difficulties caused by scarce prosodic annotation data and improving the model performance of the prosodic encoder.

[0040] In a specific implementation scenario, please refer to the following: Figure 2 x t-3 ,x t-2 ,x t-1 ,x t ,x t+1 ,x t+2 ,x t+3 ,x t+4 Each of the second sample audio frames is represented separately. Of course, in practical applications, the second sample audio frames are not limited to this. Figure 2 The illustration shows only one possible implementation in practical applications. During training, the second sample audio frame x is selected. t As the anchor sample audio frame, anchor sample audio frame x t and the second sample audio frame x before it t-3 ,x t-2 ,x t-1 Encoded by the forward coding network in the prosodic encoder, while the anchor sample audio frame x t The second sample audio frame x t+1 ,x t+2 ,x t+3 ,x t+4 Encoded by the inverse encoding network in the prosodic encoder. The second sample audio frame x mentioned above. i After encoding, the corresponding sample encoding features z can be obtained respectively. i Furthermore, the anchor sample audio frame x t and the second sample audio frame x before it t-3 ,x t-2 ,x t-1The obtained sample encoding features are sequentially encoded by the feature extraction network to obtain corresponding abstract acoustic features c t-3 ,c t-2 ,c t-1 ,c t . On this basis, the anchor sample audio frame x t can be further encoded by the rhythm encoder to obtain corresponding abstract acoustic features c t . The positive example encoding features and the negative example encoding features are selected, and the similarity between the abstract acoustic features c t and the positive example encoding features and the negative example encoding features is obtained to obtain the training loss L infonce of the rhythm encoder:

[0041]

[0042] In the above formula (1), denotes the positive example encoding features, denotes the negative example encoding features, denotes a set composed of the negative example encoding features, and z j denotes the jth negative example encoding feature in the set, and f(c t ,z t ) = exp(c t ·z t ) denotes the similarity between c t and z t , which can be calculated by dot product, which is not limited herein. On this basis, the network parameters of the rhythm encoder can be adjusted based on the training loss. For example, the network parameters of the rhythm encoder can be adjusted by gradient descent and the like. For details of the adjustment process, please refer to the technical details of gradient descent and the like, which will not be described herein. The above method can force the abstract acoustic features c t to be as close as possible to the positive example encoding features and as far as possible from the negative example encoding features

[0043] In a specific implementation scenario, for the positive example encoding features, the second sample audio frame can be selected as the anchor sample audio frame, and the sample encoding features obtained by encoding the second sample audio frame within the target frame number after the anchor sample audio frame by the reverse encoding network can be selected as the positive example encoding features, and the target frame number is the ratio used by the rhythm encoder for downsampling. For example, if the downsampling ratio of the rhythm encoder currently being trained is N times of the original frame rate, then the anchor sample audio frame x tAfterwards, at least one second sample audio frame is selected within N frames, and sample encoding features obtained by reverse encoding network coding of the second sample audio frame are obtained as positive example encoding features. In the above manner, sample encoding features obtained by reverse encoding network coding of the second sample audio frame within the target frame number after the anchor point sample audio frame are selected as positive example encoding features, and the target frame number is the rate used by the prosody encoder in downsampling, which can supervise the correlation between adjacent phonetic segments through the positive example encoding features, and help improve the accuracy of prosody modeling.

[0044] In a specific implementation scenario, for negative example encoding features, at least local negative example encoding features can be included. Specifically, a second sample audio frame can be selected as an anchor point sample audio frame, and sample encoding features obtained by reverse encoding network coding of the second sample audio frame after the target frame number after the anchor point sample audio frame are selected as negative example encoding features, and the target frame number is the rate used by the prosody encoder in downsampling. Illustratively, the downsampling rate of the prosody encoder currently being trained corresponds to N times the original frame rate, and then the anchor point sample audio frame x t Afterwards, at least one second sample audio frame is selected within N frames, and sample encoding features obtained by reverse encoding network coding of the second sample audio frame are obtained as positive example encoding features. In the above manner, sample encoding features obtained by reverse encoding network coding of the second sample audio frame within the target frame number after the anchor point sample audio frame are selected as positive example encoding features, and the target frame number is the rate used by the prosody encoder in downsampling, which can supervise the correlation between adjacent phonetic segments through the positive example encoding features, and help improve the accuracy of prosody modeling.

[0045] In a specific implementation scenario, for negative example encoding features, at least global negative example encoding features can be included. Specifically, a second sample speech belonging to the same batch as the currently used second sample speech can be selected as a target sample speech, and sample encoding features obtained by prosody encoder coding of at least one second sample audio frame in the target sample speech are obtained as negative example encoding features. Illustratively, the kth second sample speech in the batch to which the currently used second sample speech belongs can be selected as a target sample speech, and sample encoding features obtained by prosody encoder coding of at least one second sample audio frame in the target sample speech are obtained as negative example encoding features. In the above manner, the prosody features extracted from different speakers can be diversified as much as possible, which helps improve the accuracy of prosody modeling.

[0046] In a specific implementation scenario, as described above, prosody encoders corresponding to 8 times, 32 times, and 64 times the original frame rate can be constructed, which can be denoted as CPC8, CPC32, and CPC64, respectively, and used to extract prosody features at the phoneme level, the word level, and the sentence level, respectively.

[0047] Step S14: For the prosodic feature sequence corresponding to each down-sampling rate, the first phoneme feature sequence and the prosodic feature sequence are fused based on the first phoneme feature sequence to obtain a second phoneme feature sequence.

[0048] Specifically, for the prosodic feature sequence corresponding to each down-sampling rate, the first phoneme feature sequence and the prosodic feature sequence can be fused based on the cross-attention mechanism to obtain the second phoneme feature sequence. That is, the second phoneme feature sequence also contains the phoneme features of each phoneme in the phoneme sequence, and at this time the phoneme features contain not only the phoneme pronunciation related feature information, but also the phoneme prosody related feature information. For example, for the prosodic feature sequence corresponding to the i-th down-sampling rate, the key matrix of the cross-attention mechanism can be used to convert it to obtain a key feature sequence, the value matrix of the cross-attention mechanism can be used to convert it to obtain a value feature sequence, and the query matrix of the cross-attention mechanism can be used to convert the first phoneme feature sequence to obtain a query feature sequence. On this basis, the attention matrix can be calculated according to the key feature sequence and the query feature sequence, and the value feature sequence is applied to the attention matrix to obtain the second phoneme feature sequence. For specific processes, please refer to the related details of the cross-attention mechanism, which will not be described here.

[0049] Step S15: Based on each second phoneme feature sequence, a pronunciation score of the test speech is obtained.

[0050] In one implementation scenario, the phoneme features belonging to the same phoneme in various second phoneme sequences can be fused (such as addition, splicing, etc.) to obtain fused features corresponding to the phoneme, and then the fused features are used for prediction to obtain the pronunciation score of the phoneme. On this basis, the pronunciation score of the test speech can be obtained by numerically calculating (such as taking the average value, taking the maximum value, taking the minimum value, taking the median value, etc.) the pronunciation scores of each phoneme.

[0051] In one specific implementation scenario, a score prediction network can be pre-trained, and the score prediction network can include a full connection layer and a normalization layer (such as sigmoid), the full connection layer is used to predict the pronunciation score of the phoneme based on the fused features, and the normalization layer is used to normalize the pronunciation score predicted by the full connection layer to the range of 0 to 1. On this basis, the pronunciation score of the test speech can be obtained by numerically calculating the pronunciation scores of each phoneme.

[0052] In one specific implementation scenario, in order to improve the accuracy of the score prediction network, the sample data used for training the score prediction network can be labeled with sample scores, so that a loss function such as mean square error can be used to measure the difference between the sample scores and the predicted scores predicted by the score prediction network, to obtain the training loss of the score prediction network, so that the network parameters of the score prediction network can be adjusted based on the training loss.

[0053] In another implementation scenario, different from the foregoing implementation, in order to further improve the accuracy of the pronunciation detection, the acoustic features (such as Fbank, MFCC, etc.) of each audio frame can be encoded to obtain a content feature sequence before prediction based on each second phoneme feature sequence. It should be noted that the content feature sequence contains the content features of each audio frame, and the content features can specifically contain feature information related to the pronunciation content of the audio frame. On this basis, the first phoneme feature sequence and the content feature sequence can be fused to obtain a third phoneme feature sequence, so that the pronunciation scores of the test speech can be obtained based on each second phoneme feature sequence and the third phoneme feature sequence. The above-mentioned method encodes the acoustic features of each audio frame to obtain a content feature sequence, and then fuses the first phoneme feature sequence and the content feature sequence to obtain a third phoneme feature sequence, and then predicts based on each second phoneme feature sequence and the third phoneme feature sequence to obtain the pronunciation scores of the test speech. This can combine the pronunciation content and the pronunciation rhythm to perform pronunciation detection, which helps to further improve the accuracy of pronunciation detection.

[0054] In one specific implementation scenario, the first phoneme feature sequence and the content feature sequence can be fused to obtain a third phoneme feature sequence based on a cross-attention mechanism. For specific processes, please refer to the foregoing process of obtaining the second phoneme feature sequence, which will not be described here. It should be noted that the third phoneme feature sequence also contains the phoneme features of each phoneme in the phoneme sequence, except that the phoneme features now contain not only phoneme pronunciation-related feature information but also pronunciation content-related feature information.

[0055] In one specific implementation scenario, the phoneme features belonging to the same phoneme in each second phoneme feature sequence and the third phoneme feature sequence can be fused to obtain a fusion feature corresponding to the phoneme, and then the fusion feature of the phoneme is used for prediction to obtain the pronunciation score of the phoneme, so that the pronunciation score of the test speech can be obtained based on the numerical statistics of the pronunciation scores of each phoneme. For specific processes, please refer to the foregoing related description, which will not be described here. The above-mentioned method can perform pronunciation detection at the phoneme level and statistically obtain the pronunciation score of the test speech, which helps to further improve the refinement degree of pronunciation detection and improve the accuracy of pronunciation detection.

[0056] In a specific implementation scenario, the sequence of content features is obtained by encoding acoustic features by an acoustic encoder. The acoustic encoder can specifically adopt a network structure such as Conformer, Transformer, etc., without limitation. In order to improve the encoding effect of the acoustic encoder, the acoustic encoder can be pre-trained. Specifically, a first sample speech can be pre-obtained, the first sample speech is labeled with a sample phoneme sequence, and the sample speech includes a plurality of first sample audio frames. Then, acoustic features of each first sample audio frame are encoded based on the acoustic encoder to obtain a sequence of sample content features. A predicted phoneme sequence is obtained based on the sequence of sample content features. Then, the network parameters of the acoustic encoder can be adjusted based on the difference between the sample phoneme sequence and the predicted phoneme sequence. It should be noted that the specific measurement method of the difference can refer to the technical details of the loss function such as CTC, which will not be described here. The above method trains the acoustic encoder on the phoneme recognition task. On the one hand, the first sample speech labeled with the sample phoneme sequence is relatively easy to obtain, thereby helping to reduce the training difficulty. On the other hand, training the acoustic encoder on the phoneme recognition task can also force the acoustic encoder to pay attention to the feature information related to the pronunciation content as much as possible, and weaken the attention to the timbre and other aspects unrelated to the pronunciation content as much as possible, thereby helping to improve the encoding effect of the acoustic encoder.

[0057] In one implementation scenario, please refer to Figure 3 and Figure 4 , Figure 3 is a process schematic diagram of an embodiment of the pronunciation detection method of the present application, Figure 4 is a process schematic diagram of another embodiment of the pronunciation detection method of the present application. As shown in the figure, the multi-level prosody encoder (i.e., the prosody encoder corresponding to different down-sampling rates, such as CPC8, CPC32, CPC64, etc.) can encode the test speech to obtain the sequence of prosodic features corresponding to different down-sampling rates. At the same time, the acoustic encoder can encode the test speech to obtain the sequence of content features. In addition, the decoder (such as the Transformer decoder) can extract the first phoneme feature sequence of the phoneme sequence. On this basis, the first phoneme feature sequence can be fused with the sequence of prosodic features corresponding to different down-sampling rates based on the cross-attention mechanism to obtain the second phoneme feature sequence, and the first phoneme feature sequence can be fused with the sequence of content features based on the cross-attention mechanism to obtain the third phoneme feature sequence. Then, the phoneme features belonging to the same phoneme in each second phoneme feature sequence and third phoneme feature sequence can be fused to obtain the fusion feature corresponding to the phoneme. The pronunciation score of the phoneme can be obtained based on the fusion feature of the phoneme, and the pronunciation score of the phoneme can be normalized. Then, the pronunciation scores of the phonemes after normalization can be statistically counted to obtain the pronunciation score of the test speech.

[0058] The above scheme extracts the test speech collected by the target object reading the target text, and obtains the phoneme sequence obtained by converting the phonemes of the target text. The test speech includes a plurality of audio frames. The phoneme features of each phoneme in the phoneme sequence are extracted to obtain a first phoneme feature sequence. A plurality of down-sampling rates are determined. For each down-sampling rate, the plurality of audio frames are encoded based on the down-sampling rate to obtain a prosody feature sequence corresponding to the down-sampling rate. For the prosody feature sequences corresponding to each down-sampling rate, the first phoneme feature sequence and the prosody feature sequence are fused to obtain a second phoneme feature sequence. The pronunciation score of the test speech is obtained by predicting based on each second phoneme feature sequence. Since the prosody feature sequences corresponding to each down-sampling rate are obtained by encoding with different down-sampling rates, the pronunciation detection can be assisted by multiple levels of prosody features, which helps to improve the refinement of pronunciation detection. Therefore, the accuracy of pronunciation detection can be improved.

[0059] Please refer to Figure 5 , Figure 5 is a schematic diagram of an embodiment of a pronunciation detection device 50 of the present application. The pronunciation detection device 50 includes an acquisition module 51, an extraction module 52, a determination module 53, an encoding module 54, a fusion module 55, and a prediction module 56. The acquisition module 51 is configured to acquire test speech collected by a target object reading a target text, and to acquire a phoneme sequence obtained by converting the phonemes of the target text. The test speech includes a plurality of audio frames. The extraction module 52 is configured to extract phoneme features of each phoneme in the phoneme sequence to obtain a first phoneme feature sequence. The determination module 53 is configured to determine a plurality of down-sampling rates. The encoding module 54 is configured to, for each down-sampling rate, encode the plurality of audio frames based on the down-sampling rate to obtain a prosody feature sequence corresponding to the down-sampling rate. The fusion module 55 is configured to, for the prosody feature sequences corresponding to each down-sampling rate, fuse the first phoneme feature sequence and the prosody feature sequence to obtain a second phoneme feature sequence. The prediction module 56 is configured to predict based on each second phoneme feature sequence to obtain a pronunciation score of the test speech.

[0060] The above scheme, since the prosody feature sequences corresponding to each down-sampling rate are obtained by encoding with different down-sampling rates, the pronunciation detection can be assisted by multiple levels of prosody features, which helps to improve the refinement of pronunciation detection. Therefore, the accuracy of pronunciation detection can be improved.

[0061] In some disclosed embodiments, the encoding module 54 is further configured to encode the acoustic features of the audio frames to obtain a content feature sequence; the fusion module 55 is further configured to fuse the first phoneme feature sequence and the content feature sequence to obtain a third phoneme feature sequence; and the prediction module 56 is specifically configured to predict based on the second phoneme feature sequence and the third phoneme feature sequence to obtain the pronunciation score of the test speech.

[0062] Therefore, the encoding based on the acoustic features of the audio frames to obtain the content feature sequence, the fusion based on the first phoneme feature sequence and the content feature sequence to obtain the third phoneme feature sequence, and the prediction based on the second phoneme feature sequence and the third phoneme feature sequence to obtain the pronunciation score of the test speech can combine pronunciation content and pronunciation prosody for pronunciation detection, which helps to further improve the accuracy of pronunciation detection.

[0063] In some disclosed embodiments, the prediction module 56 includes a phoneme feature fusion sub-module configured to fuse the phoneme features belonging to the same phoneme in the second phoneme feature sequence and the third phoneme feature sequence to obtain a fusion feature of the corresponding phoneme; the prediction module 56 includes a phoneme score prediction sub-module configured to predict based on the fusion feature of the phoneme to obtain a pronunciation score of the phoneme; and the prediction module 56 includes a numerical statistics sub-module configured to statistically analyze the pronunciation scores of the phonemes to obtain the pronunciation score of the test speech.

[0064] Therefore, pronunciation detection can be performed at the phoneme level, and the pronunciation score of the test speech can be statistically obtained based on this, which helps to further improve the refinement degree of pronunciation detection and improve the accuracy of pronunciation detection.

[0065] In some disclosed embodiments, the content feature sequence is obtained by encoding the acoustic features by an acoustic encoder, the pronunciation detection apparatus 50 further includes a sample acquisition module configured to acquire a first sample speech; the first sample speech is labeled with a sample phoneme sequence, and the sample speech includes a plurality of first sample audio frames; the pronunciation detection apparatus 50 further includes a sample encoding module configured to encode the acoustic features of the first sample audio frames based on the acoustic encoder to obtain a sample content feature sequence; the pronunciation detection apparatus 50 further includes a sequence prediction module configured to predict based on the sample content feature sequence to obtain a predicted phoneme sequence; and the pronunciation detection apparatus 50 further includes a parameter adjustment module configured to adjust the network parameters of the acoustic encoder based on the difference between the sample phoneme sequence and the predicted phoneme sequence.

[0066] Therefore, by training the acoustic encoder on the phoneme recognition task, on the one hand, the first sample speech with the annotated phoneme sequence is relatively easy to obtain, thereby being able to help reduce the training difficulty, and on the other hand, training the acoustic encoder on the phoneme recognition task is also able to force the acoustic encoder to pay attention to the feature information related to the pronunciation content as much as possible, and to weaken the attention to the tone and other aspects irrelevant to the pronunciation content as much as possible, thereby being able to help improve the encoding effect of the acoustic encoder.

[0067] In some disclosed embodiments, the prosody feature sequences corresponding to different down-sampling rates are respectively obtained by encoding a plurality of audio frames by the prosody encoders corresponding to the down-sampling rates, and the prosody encoders are trained in a contrastive predictive coding manner based on the second sample speech including a plurality of second sample audio frames.

[0068] Therefore, the prosody feature sequences corresponding to different down-sampling rates are respectively obtained by encoding a plurality of audio frames by the prosody encoders corresponding to the down-sampling rates, and the prosody encoders are trained in a contrastive predictive coding manner based on the second sample speech including a plurality of second sample audio frames, so that the prosody encoders can be trained in an unsupervised manner, thereby being able to train the prosody encoders without relying on sample speech data with prosody annotations, and further being able to greatly alleviate the problems caused by the scarcity of prosody annotation data for model training, thereby being able to help improve the model performance of the prosody encoders.

[0069] In some disclosed embodiments, the pronunciation detection apparatus 50 further includes an anchor point selection module configured to select a second sample audio frame as an anchor point sample audio frame; wherein the anchor point sample audio frame and the second sample audio frames before the anchor point sample audio frame are encoded by the forward encoding network in the prosody encoder, and the second sample audio frames after the anchor point sample audio frame are encoded by the backward encoding network in the prosody encoder; and the pronunciation detection apparatus 50 further includes a positive example selection module configured to select the sample encoding features of the second sample audio frames within a target frame number after the anchor point sample audio frame and encoded by the backward encoding network as positive example encoding features; wherein the target frame number is the down-sampling rate used by the prosody encoder.

[0070] Therefore, the sample encoding features of the second sample audio frames within the target frame number after the anchor point sample audio frame and encoded by the backward encoding network are selected as the positive example encoding features, and the target frame number is the down-sampling rate used by the prosody encoder, so that the positive example encoding features can be used to supervise the modeling of the relevance between adjacent segments, thereby being able to help improve the accuracy of prosody modeling.

[0071] In some disclosed embodiments, the pronunciation detection apparatus 50 further comprises a negative example selection module configured to select, as the negative example encoding features, the sample encoding features of the second sample audio frame after the target frame number of the anchor point sample audio frame and encoded by the reverse encoding network.

[0072] Therefore, by selecting the second sample audio frame as the anchor point sample audio frame and selecting, as the negative example encoding features, the sample encoding features of the second sample audio frame after the target frame number of the anchor point sample audio frame and encoded by the reverse encoding network, and by taking the target frame number as the downsampling rate of the prosody encoder, the correlation between adjacent phonetic segments can be supervised by the negative example encoding features, which helps to improve the accuracy of prosody modeling.

[0073] In some disclosed embodiments, the pronunciation detection apparatus 50 further comprises a target selection module configured to select, as the target sample speech, the second sample speech belonging to the same batch as the current second sample speech, and further comprises a negative example acquisition module configured to acquire, as the negative example encoding features, the sample encoding features of at least one second sample audio frame in the target sample speech and encoded by the prosody encoder.

[0074] Therefore, the prosody features extracted from different speakers can be diversified as much as possible, which helps to improve the accuracy of prosody modeling.

[0075] In some disclosed embodiments, the acquisition module 51 is specifically configured to align the target text with the test speech to obtain the phoneme sequence.

[0076] Therefore, by aligning the target text with the test speech to obtain the phoneme sequence, the influence of one word with multiple pronunciations on phoneme conversion can be alleviated as much as possible, which helps to improve the accuracy of phoneme conversion.

[0077] In some disclosed embodiments, the determination module 53 is specifically configured to multiply the frame rate of the test speech by a plurality of rates to obtain a plurality of downsampling rates.

[0078] Therefore, by multiplying the frame rate of the test speech by a plurality of rates to obtain a plurality of downsampling rates, the accuracy of modeling prosody at multiple levels such as phoneme level, word level, and sentence level can be improved.

[0079] For reference Figure 6 , Figure 6is a schematic diagram of a framework of an embodiment of the electronic device 60 of the present application. The electronic device 60 comprises a memory 61 and a processor 62 coupled with each other, the memory 61 stores program instructions, and the processor 62 is configured to execute the program instructions to implement the steps in any of the above-mentioned pronunciation detection method embodiments. Specifically, the electronic device 60 can include but is not limited to a desktop computer, a notebook computer, a server, a mobile phone, a tablet computer, and the like, which are not limited herein.

[0080] Specifically, the processor 62 is configured to control itself and the memory 61 to implement the steps in any of the above-mentioned pronunciation detection method embodiments. The processor 62 can also be referred to as a CPU (Central Processing Unit). The processor 62 can be an integrated circuit chip with processing capability. The processor 62 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0081] The above-mentioned scheme, the electronic device 60 can implement the steps in any of the above-mentioned pronunciation detection method embodiments, and since the prosodic feature sequences corresponding to various down-sampling rates are obtained by encoding at different down-sampling rates, the pronunciation detection can be assisted by multiple levels of prosodic features, which helps to improve the fine degree of pronunciation detection. Therefore, the accuracy of pronunciation detection can be improved.

[0082] Please refer to Figure 7 , Figure 7 is a schematic diagram of a framework of an embodiment of the computer readable storage medium 70 of the present application. The computer readable storage medium 70 stores program instructions 71 executable by a processor, and the program instructions 71 are configured to implement the steps in any of the above-mentioned pronunciation detection method embodiments.

[0083] The above-mentioned scheme, the computer readable storage medium 70 can implement the steps in any of the above-mentioned pronunciation detection method embodiments, and since the prosodic feature sequences corresponding to various down-sampling rates are obtained by encoding at different down-sampling rates, the pronunciation detection can be assisted by multiple levels of prosodic features, which helps to improve the fine degree of pronunciation detection. Therefore, the accuracy of pronunciation detection can be improved.

[0084] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, it will not be repeated here.

[0085] The above description of various embodiments tends to emphasize the differences between various embodiments, and the same or similar parts can be referred to each other. For brevity, it will not be repeated here.

[0086] In several embodiments provided in the present application, it should be understood that the disclosed methods and apparatuses can be implemented in other ways. For example, the above-described apparatus implementation is only schematic, for example, the division of modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed each other can be indirect coupling or communication connection through some interface, apparatus or unit, which can be electrical, mechanical or other form.

[0087] The unit described as a separate component can or can not be physically separated, and the component shown as a unit can or can not be a physical unit, that is, it can be located in one place, or it can be distributed to a plurality of network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the present embodiment scheme.

[0088] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0089] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the methods in the embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various other media that can store program codes.

[0090] If the technical solutions of the present application involve personal information, the product applying the technical solutions of the present application has been informed of the personal information processing rules before processing the personal information, and has obtained the personal independent consent. If the technical solutions of the present application involve sensitive personal information, the product applying the technical solutions of the present application has obtained the personal independent consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as camera, a clear and prominent mark is set to inform that the personal information collection range has been entered, and the personal information will be collected. If the person voluntarily enters the collection range, it is regarded as agreeing to collect the personal information. Or, on the device for processing personal information, the personal information processing rules are informed by using obvious mark / information, and the personal authorization is obtained by means of pop-up information or asking the person to upload the personal information. The personal information processing rules can include personal information processor, personal information processing purpose, processing method, and personal information type, etc.

Claims

1. A method of detecting a sound production, characterized by, The method comprises: obtaining test speech collected by a test subject reading a target text, and obtaining a phoneme sequence obtained by phoneme conversion of the target text; wherein the test speech comprises a plurality of audio frames; extracting phoneme features of each phoneme in the phoneme sequence to obtain a first phoneme feature sequence, and determining a plurality of down-sampling rates; for each of the down-sampling rates, encoding the plurality of audio frames based on the down-sampling rate to obtain a prosody feature sequence corresponding to the down-sampling rate; for the prosody feature sequences corresponding to each of the down-sampling rates, respectively, fusing the first phoneme feature sequence and the prosody feature sequence to obtain a second phoneme feature sequence; based on each of the second phoneme feature sequences, predicting to obtain a pronunciation score of the test speech.

2. The method of claim 1, wherein, Before the step of predicting based on each of the second phoneme feature sequences to obtain a pronunciation score of the test speech, the method further comprises: encoding acoustic features of each of the audio frames to obtain a content feature sequence; fusing the first phoneme feature sequence and the content feature sequence to obtain a third phoneme feature sequence; the step of predicting based on each of the second phoneme feature sequences to obtain a pronunciation score of the test speech comprises: predicting based on each of the second phoneme feature sequences and the third phoneme feature sequence to obtain a pronunciation score of the test speech.

3. The method of claim 2, wherein, the step of predicting based on each of the second phoneme feature sequences and the third phoneme feature sequence to obtain a pronunciation score of the test speech comprises: fusing phoneme features belonging to the same phoneme in each of the second phoneme feature sequences and the third phoneme feature sequence to obtain a fusion feature corresponding to the phoneme; predicting based on the fusion feature of the phoneme to obtain a pronunciation score of the phoneme; based on each of the pronunciation scores of the phonemes, performing numerical statistics to obtain a pronunciation score of the test speech.

4. The method of claim 2, wherein, The content feature sequence is obtained by encoding the acoustic features by an acoustic encoder, and the training step of the acoustic encoder comprises: obtaining a first sample speech; wherein the first sample speech is labeled with a sample phoneme sequence, and the sample speech comprises a plurality of first sample audio frames; encoding acoustic features of each of the first sample audio frames based on the acoustic encoder to obtain a sample content feature sequence; based on the sample content feature sequence, predicting to obtain a predicted phoneme sequence; based on the difference between the sample phoneme sequence and the predicted phoneme sequence, adjusting the network parameters of the acoustic encoder.

5. The method of claim 1, wherein, The prosody feature sequences corresponding to different down-sampling rates are obtained by encoding the plurality of audio frames by a prosody encoder corresponding to the down-sampling rate, and the prosody encoder is trained by a contrastive prediction coding method based on a second sample speech comprising a plurality of second sample audio frames.

6. The method of claim 5, wherein, During the training process of the prosody encoder by the contrastive prediction coding method, the acquisition step of positive example encoding features comprises: select the second sample audio frame as an anchor point sample audio frame; wherein the anchor point sample audio frame and the second sample audio frames before the anchor point sample audio frame are encoded by a forward encoding network in the prosody encoder, and the second sample audio frames after the anchor point sample audio frame are encoded by a reverse encoding network in the prosody encoder; select the sample encoding features of the second sample audio frames within a target frame number after the anchor point sample audio frame and encoded by the reverse encoding network as positive example encoding features; wherein the target frame number is a rate used by the prosody encoder when down-sampling.

7. The method of claim 5, wherein, In the training process of the prosody encoder using the contrastive prediction encoding, the step of obtaining negative example encoding features includes: select the second sample audio frame as an anchor point sample audio frame; wherein the anchor point sample audio frame and the second sample audio frames before the anchor point sample audio frame are encoded by a forward encoding network in the prosody encoder, and the second sample audio frames after the anchor point sample audio frame are encoded by a reverse encoding network in the prosody encoder; select the sample encoding features of the second sample audio frames after a target frame number after the anchor point sample audio frame and encoded by the reverse encoding network as negative example encoding features; wherein the target frame number is a rate used by the prosody encoder when down-sampling.

8. The method of claim 5, wherein, In the training process of the prosody encoder using the contrastive prediction encoding, the step of obtaining negative example encoding features includes: select the second sample audio frame as an anchor point sample audio frame; wherein the anchor point sample audio frame and the second sample audio frames before the anchor point sample audio frame are encoded by a forward encoding network in the prosody encoder, and the second sample audio frames after the anchor point sample audio frame are encoded by a reverse encoding network in the prosody encoder; select the sample encoding features of the second sample audio frames within a target frame number after the anchor point sample audio frame and encoded by the reverse encoding network as positive example encoding features; 9. The method of claim 1, wherein, wherein the target frame number is a rate used by the prosody encoder when down-sampling. In the training process of the prosody encoder using the contrastive prediction encoding, the step of obtaining negative example encoding features includes:

10. The method of claim 1, wherein, select the second sample audio frame as an anchor point sample audio frame; wherein the anchor point sample audio frame and the second sample audio frames before the anchor point sample audio frame are encoded by a forward encoding network in the prosody encoder, and the second sample audio frames after the anchor point sample audio frame are encoded by a reverse encoding network in the prosody encoder; select the sample encoding features of the second sample audio frames within a target frame number after the anchor point sample audio frame and encoded by the reverse encoding network as positive example encoding features; 11. A sound production detection apparatus characterized by comprising: wherein the target frame number is a rate used by the prosody encoder when down-sampling. In the training process of the prosody encoder using the contrastive prediction encoding, the step of obtaining negative example encoding features includes: select the second sample audio frame as an anchor point sample audio frame; wherein the anchor point sample audio frame and the second sample audio frames before the anchor point sample audio frame are encoded by a forward encoding network in the prosody encoder, and the second sample audio frames after the anchor point sample audio frame are encoded by a reverse encoding network in the prosody encoder; select the sample encoding features of the second sample audio frames within a target frame number after the anchor point sample audio frame and encoded by the reverse encoding network as positive example encoding features; wherein the target frame number is a rate used by the prosody encoder when down-sampling. In the training process of the prosody encoder using the contrastive prediction encoding, the step of obtaining negative example encoding features includes: select the second sample audio frame as an anchor point sample audio frame; wherein the anchor point sample audio frame and the second sample audio frames before the anchor point sample audio frame are encoded by a forward encoding network in the prosody encoder, and the second sample audio frames after the anchor point sample audio frame are encoded by a reverse encoding network in the prosody encoder; select the sample encoding features of the second sample audio frames within a target frame number after the anchor point sample audio frame and encoded by the reverse encoding network as positive example encoding features; wherein the target frame number is a rate used by the prosody encoder when down-sampling. In the training process of the prosody encoder using the contrastive prediction encoding, the step of obtaining negative example encoding features includes: select the second sample audio frame as an anchor point sample audio frame; wherein the anchor point sample audio frame and the second sample audio frames before the anchor point sample audio frame are encoded by a forward encoding network in the prosody encoder, and the second sample audio frames after the anchor point sample audio frame are encoded by a reverse encoding network in the prosody encoder; select the sample encoding features of the second sample audio frames within a target frame number after the anchor point sample audio frame and encoded by the reverse encoding network as positive example encoding features; wherein the target frame number is a rate used by the prosody encoder when down-sampling. In the training process of the prosody encoder using the contrastive prediction encoding, the step of obtaining negative example encoding features includes: select the second sample audio frame as an anchor point sample audio frame; wherein the anchor point sample audio frame and the second sample audio frames before the anchor point sample audio frame are encoded by a forward encoding network in the prosody encoder, and the second sample audio frames after the anchor point sample audio frame are encoded by a reverse encoding network in the prosody encoder; select the sample encoding features of the second sample audio frames within a target frame number after the anchor point sample audio frame and encoded by the reverse encoding network as positive example encoding features; wherein the target frame number is a rate used by the prosody encoder when down-sampling. In the training process of the prosody encoder using the contrastive prediction encoding, the step of obtaining negative example encoding features includes: select the second sample audio frame as an anchor point sample audio frame; wherein the anchor point sample audio frame and the second sample audio frames before the anchor point sample audio frame are encoded by a forward encoding network in the prosody encoder, and the second sample audio frames after the anchor point sample audio frame are encoded by a reverse encoding network in the prosody encoder; select the sample encoding features of the second sample audio frames within a target frame number after the anchor point sample audio frame and encoded by the reverse encoding network as positive example encoding features; wherein the target frame number is a rate used by the prosody encoder when down-sampling. In the training process of the prosody encoder using the contrastive prediction encoding, the step of obtaining negative example encoding features includes: select the second sample audio frame as an anchor point sample audio frame; wherein the anchor point sample audio frame and the second sample audio frames before the anchor point sample audio frame are encoded by a forward encoding network in the prosody encoder, and the second sample audio frames after the anchor point sample audio frame are encoded by a reverse encoding network in the prosody encoder; select the sample encoding features of the second sample audio frames within a target frame number after the anchor point sample audio frame and encoded by the reverse encoding network as positive example encoding features; wherein the target frame number is a rate used by the prosody encoder when down-sampling. In the training process of the prosody encoder using the contrastive prediction encoding, the step of obtaining negative example encoding features includes: select the second sample audio frame as an anchor point sample audio frame; wherein the anchor point sample audio frame and the second sample audio frames before the anchor point sample audio frame are encoded by a forward encoding network in the prosody encoder, and the second sample audio frames after the anchor point sample audio frame are encoded by a reverse encoding network in the prosody encoder; select the sample encoding features of the second sample audio frames within a target frame number after the anchor point sample audio frame and encoded by the reverse encoding network as positive example encoding features; wherein the target frame number is a rate used by the prosody encoder when down-sampling. In the training process of the prosody encoder using the contrastive prediction encoding, the step of obtaining negative example encoding features includes: select the second sample audio frame as an anchor point sample audio frame; wherein the anchor point sample audio frame and the second sample audio frames before the anchor point sample audio frame are encoded by a forward encoding network in the prosody encoder, and the second sample audio frames after the anchor point sample audio frame are encoded by a reverse encoding network in the prosody encoder; select the sample encoding features of the second sample audio frames within a target frame number after the anchor point sample audio frame and encoded by the reverse encoding network as positive example encoding features; wherein the target frame number is a rate used by the prosody encoder when down-sampling. In the training process of the prosody encoder using the contrastive prediction encoding, the step of obtaining negative example encoding features includes: select the second sample audio frame as an anchor point sample audio frame; wherein the anchor point sample audio frame and the second sample audio frames before the anchor point sample audio frame are encoded by a forward encoding network in the prosody encoder, and the second sample audio frames after the anchor point sample audio frame are encoded by a reverse encoding network in the prosody encoder; select the sample encoding features of the second sample audio frames within a target frame number after the anchor point sample audio frame and encoded by the reverse encoding network as positive example encoding features; wherein the target frame number is a rate used by the prosody encoder when down-sampling. In the training process of the prosody encoder using the contrastive prediction encoding, the step of obtaining negative example encoding features includes: select the second sample audio frame as an anchor point sample audio frame; wherein the anchor point sample audio frame and the second sample audio frames before the anchor point sample audio 12. An electronic device, comprising: A computer program product comprising a memory having program instructions stored therein and a processor configured to execute the program instructions to implement the method of any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, A computer program product comprising a memory having program instructions stored therein and a processor configured to execute the program instructions to implement the method of any one of claims 1 to 10.

Images