Speech synthesis model training method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a speech synthesis model training method and device, electronic equipment and a storage medium, belongs to the technical field of speech synthesis, and is suitable for the field of financial science and technology. The method comprises the following steps: acquiring audio and video sample data; performing voice extraction on the audio and video sample data to obtain voice data; performing audio text recognition on the voice data to obtain a voice text; performing audio attribute recognition on the voice data to obtain audio attributes; based on the multiple audio attributes, audio attribute description is carried out on the audio and video sample data, multiple single-attribute description texts are obtained, and the single-attribute description texts are used for describing one audio attribute of the audio and video sample data; based on the voice text, instantiating the single-attribute description text to obtain voice synthesis sample data; and based on the speech synthesis sample data, performing model training on a preset speech synthesis model. According to the embodiment of the invention, the accuracy of speech synthesis model training can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, applicable to the field of financial technology, and in particular to a speech synthesis model training method and device, electronic equipment and storage medium. Background Art

[0002] Speech synthesis refers to the conversion of written text into speech. For example, in the financial field, speech synthesis is performed on insurance documents to generate voice data of the insurance documents, so that users can learn the relevant content of the insurance documents through hearing.

[0003] At present, most common speech synthesis methods synthesize speech data with speech attributes based on instruction text. For example, when the instruction text is: generate a voice that sounds like a young woman happily saying "hello", the speech attributes of the speech data are young women and happiness. However, the number of instruction texts is scarce, resulting in low accuracy in speech synthesis model training. Therefore, how to improve the accuracy of speech synthesis model training has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a speech synthesis model training method and device, electronic device and storage medium, aiming to improve the accuracy of speech synthesis model training.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a speech synthesis model training method, the method comprising:

[0006] Get audio and video sample data;

[0007] Performing voice extraction on the audio and video sample data to obtain voice data;

[0008] Performing audio-to-text recognition on the voice data to obtain voice text;

[0009] Performing audio attribute recognition on the voice data to obtain audio attributes;

[0010] Based on the multiple audio attributes, the audio and video sample data is described in audio attributes to obtain multiple single attribute description texts, wherein the single attribute description text is used to describe one of the audio attributes of the audio and video sample data;

[0011] Based on the speech text, instantiate the single attribute description text to obtain the speech synthesis sample data;

[0012] Based on the speech synthesis sample data, model training is performed on a preset speech synthesis model.

[0013] In some embodiments, performing audio attribute recognition on the voice data to obtain audio attributes includes:

[0014] Based on the voice data, performing video extraction on the audio and video sample data to obtain video sample data;

[0015] Based on a preset audio attribute extraction model, extract attributes from the speech data to obtain a first audio attribute and a first attribute confidence;

[0016] Performing face recognition on the video sample data to obtain a face image;

[0017] Based on a preset face attribute extraction model and the face image, perform attribute recognition on the voice data to obtain a second audio attribute and a second attribute confidence;

[0018] The audio attribute is generated based on the first audio attribute, the first attribute confidence, the second audio attribute, and the second attribute confidence.

[0019] In some embodiments, the extracting attributes of the speech data based on a preset audio attribute extraction model to obtain the first audio attribute and the first attribute confidence level includes:

[0020] Extracting features from the speech data to obtain audio fundamental frequency, Mel-spectrum cepstral coefficients, pitch features, and volume features;

[0021] Based on the audio attribute extraction model and the audio fundamental frequency, the speaker gender recognition is performed on the speech data to obtain the speaker gender attribute and the audio gender attribute confidence of the speaker gender attribute;

[0022] Based on the audio attribute extraction model, the Mel-spectrogram cepstral coefficients and the pitch features, extracting emotional features from the speech data to obtain audio emotional attributes and audio emotional attribute confidences of the audio emotional attributes;

[0023] Based on the audio attribute extraction model, the tone feature, and the volume feature, extracting speech rate features from the speech data to obtain an audio speech rate attribute and an audio speech rate attribute confidence of the audio speech rate attribute;

[0024] Merging the speaker gender attribute, the audio emotion attribute, and the audio speech speed attribute to obtain the first audio attribute;

[0025] The audio gender attribute confidence, the audio emotion attribute confidence, and the audio speaking speed attribute confidence are aggregated to obtain the first attribute confidence.

[0026] In some embodiments, the performing attribute recognition on the voice data based on the position probability, based on a preset face attribute extraction model and the face image, to obtain a second audio attribute and a second attribute confidence level includes:

[0027] Performing gender recognition on the face image based on the face attribute extraction model to obtain a face gender attribute and a face gender attribute confidence level of the face gender attribute;

[0028] Performing emotion recognition on the face image based on the face attribute extraction model to obtain face emotion attributes and face emotion attribute confidence levels of the face emotion attributes;

[0029] Performing age prediction on the face image based on the face attribute extraction model to obtain a face age attribute and a face age attribute confidence level of the face age attribute;

[0030] Merging the face gender attribute, the face emotion attribute and the face age attribute to obtain the second audio attribute;

[0031] The face gender attribute confidence, the face emotion attribute confidence and the face age attribute confidence are combined to obtain the second attribute confidence.

[0032] In some embodiments, generating the audio attribute based on the first audio attribute, the first attribute confidence, the second audio attribute, and the second attribute confidence includes:

[0033] Comparing the first audio attribute with the second audio attribute to obtain the same audio attribute and the different audio attribute, wherein the same audio attribute refers to the same attribute as the second audio attribute in the first audio attribute, and the different audio attribute refers to the different attribute between the first audio attribute and the audio attribute;

[0034] Based on the first attribute confidence and the second attribute confidence, performing attribute selection on the same audio attribute to obtain a predicted audio attribute;

[0035] The predicted audio attribute and the different audio attribute are merged to obtain the audio attribute.

[0036] In some embodiments, the audio attribute description of the audio and video sample data is performed based on the multiple audio attributes to obtain multiple single attribute description texts, including:

[0037] Based on the audio and video sample data, construct an attribute description template;

[0038] Based on the audio attribute, the attribute description template is filled to obtain the single attribute description text.

[0039] In some embodiments, the instantiation processing of the single attribute description text based on the speech text to obtain the speech synthesis sample data includes:

[0040] Merging the plurality of single-attribute description texts to obtain a speech synthesis instruction text;

[0041] The speech synthesis instruction text is concatenated with the speech text to obtain the speech synthesis sample data.

[0042] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech synthesis model training device, the device comprising:

[0043] A data acquisition module, used to acquire audio and video sample data;

[0044] A speech recognition module, used to perform speech extraction on the audio and video sample data to obtain speech data;

[0045] An audio recognition module, used to perform audio text recognition on the voice data to obtain voice text;

[0046] An attribute recognition module, used to perform audio attribute recognition on the voice data to obtain audio attributes;

[0047] An attribute description module, used for performing audio attribute description on the audio and video sample data based on the multiple audio attributes to obtain multiple single attribute description texts, wherein the single attribute description text is used to describe one of the audio attributes of the audio and video sample data;

[0048] A text instantiation module, used for instantiating the single attribute description text based on the speech text to obtain the speech synthesis sample data;

[0049] The model training module is used to perform model training on a preset speech synthesis model based on the speech synthesis sample data.

[0050] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0051] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0052] The speech synthesis model training method and device, electronic device and storage medium proposed in the present application obtain rich audio and video sample data and perform audio attribute recognition on the speech data to obtain audio attributes, and then perform audio attribute description on the audio and video sample data based on multiple audio attributes to obtain multiple single attribute description texts, thereby solving the problem of scarce instruction text. Subsequently, speech extraction is performed on the audio and video sample data to obtain speech data, and then audio text recognition is performed on the speech data to obtain speech text. Then, based on the speech text, the speech synthesis instruction text is instantiated to obtain speech synthesis sample data, and based on the speech synthesis sample data, a preset speech synthesis model is trained to ensure that the speech synthesis model receives more comprehensive sample data, thereby improving the efficiency and accuracy of speech synthesis model training and optimizing the quality of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of the speech synthesis model training method provided in an embodiment of the present application;

[0054] Figure 2 yes Figure 1 Flow chart of step S104 in FIG.

[0055] Figure 3 yes Figure 2 Flow chart of step S202 in FIG.

[0056] Figure 4 yes Figure 2 Flow chart of step S204 in FIG.

[0057] Figure 5 yes Figure 2 Flow chart of step S205 in FIG.

[0058] Figure 6 yes Figure 1 Flow chart of step S105 in FIG.

[0059] Figure 7 yes Figure 1 Flow chart of step S106 in FIG.

[0060] Figure 8 It is a structural schematic diagram of a speech synthesis model training device provided in an embodiment of the present application;

[0061] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0065] First, some nouns involved in this application are analyzed:

[0066] Support Vector Machine: Support vector machine is a powerful supervised learning algorithm, mainly used for classification and regression tasks. It separates data points of different categories in the feature space by finding an optimal hyperplane and builds a model through support vectors (sample points closest to the decision boundary). Support vector machines are particularly suitable for high-dimensional data and can effectively handle nonlinear problems using kernel functions. The algorithm has good generalization ability and can maintain high accuracy when facing complex pattern recognition tasks.

[0067] Neural network model: A neural network model is a computational model that mimics the structure of neurons in the human brain and is used to handle complex pattern recognition and prediction tasks. It consists of multiple layers of nodes or neurons, including input layers, hidden layers, and output layers, which pass information to each other through connection weights. By adjusting these weights, the neural network is able to learn patterns and features in the data, and is usually trained using a backpropagation algorithm to minimize prediction errors. Neural networks have shown excellent performance in areas such as image recognition, natural language processing, and speech recognition, especially variants such as deep neural networks and convolutional neural networks, which have promoted the development of artificial intelligence.

[0068] Speech synthesis refers to the conversion of written text into speech. For example, in the financial field, speech synthesis is performed on insurance documents to generate voice data of the insurance documents, so that users can learn the relevant content of the insurance documents through hearing.

[0069] At present, most common speech synthesis methods synthesize speech data with speech attributes based on instruction text. For example, when the instruction text is: generate a voice that sounds like a young woman happily saying "hello", the speech attributes of the speech data are young women and happiness. However, the number of instruction texts is scarce, resulting in low accuracy in speech synthesis model training. Therefore, how to improve the accuracy of speech synthesis model training has become a technical problem that needs to be solved urgently.

[0070] Based on this, the embodiments of the present application provide a speech synthesis model training method and device, an electronic device and a storage medium, aiming to improve the accuracy of speech synthesis model training.

[0071] The speech synthesis model training method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the speech synthesis model training method in the embodiments of the present application is described.

[0072] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0073] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0074] The speech synthesis model training method provided in the embodiment of the present application relates to the field of speech synthesis technology and is applicable to the field of financial technology. The speech synthesis model training method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech synthesis model training method, etc., but is not limited to the above forms.

[0075] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0076] Figure 1 is an optional flowchart of the speech synthesis model training method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0077] Step S101, obtaining audio and video sample data;

[0078] Step S102, performing voice extraction on the audio and video sample data to obtain voice data;

[0079] Step S103, performing audio text recognition on the voice data to obtain voice text;

[0080] Step S104, performing audio attribute recognition on the voice data to obtain audio attributes;

[0081] Step S105, based on the multiple audio attributes, the audio and video sample data is described in audio attributes to obtain multiple single attribute description texts, wherein the single attribute description text is used to describe one of the audio attributes of the audio and video sample data;

[0082] Step S106, based on the speech text, instantiate the single attribute description text to obtain speech synthesis sample data;

[0083] Step S107: performing model training on a preset speech synthesis model based on the speech synthesis sample data.

[0084] Steps S101 to S107 shown in the embodiment of the present application are to perform speech extraction on the acquired audio and video sample data to obtain speech data, and then perform audio text recognition on the speech data to obtain speech text. Secondly, audio attribute recognition is performed on the audio and video sample data to obtain audio attributes. Then, based on the multiple audio attributes, the audio attribute description of the audio and video sample data is performed to obtain multiple single attribute description texts for describing one of the audio attributes of the audio and video sample data. Furthermore, based on the speech text, the single attribute description text is instantiated to obtain speech synthesis sample data. Finally, based on the speech synthesis sample data, a preset speech synthesis model is trained. Therefore, the present application solves the problem of scarce instruction text by acquiring rich audio and video sample data and performing audio attribute recognition on the voice data to obtain audio attributes, and then performing audio attribute description on the audio and video sample data based on multiple audio attributes to obtain multiple single attribute description texts. Subsequently, voice extraction is performed on the audio and video sample data to obtain voice data, and then audio text recognition is performed on the voice data to obtain voice text. Based on the voice text, the voice synthesis instruction text is instantiated to obtain voice synthesis sample data, and based on the voice synthesis sample data, a preset voice synthesis model is trained to ensure that the voice synthesis model receives more comprehensive sample data, thereby improving the efficiency and accuracy of the voice synthesis model training and optimizing the quality of voice synthesis.

[0085] In step S101 of some embodiments, the audio and video sample data refers to the audio and video data generated by audio and video recording of the speaker. It should be noted that the audio and video sample data includes the speaker's voice data and the speaker's face data.

[0086] The present application can obtain audio and video sample data containing audio data of the speaker's voice and video data of the speaker's facial expression by performing audio and video recording of the speaker's facial expression during the speaking process.

[0087] In step S102 of some embodiments, a video processing library can be used to extract an audio stream, i.e., an audio file, from the audio and video sample data, and then the audio processing library is used to read the audio file to obtain an audio signal. Furthermore, the audio signal is framed to obtain an audio frame, and a window function is applied to each audio frame to facilitate the conversion of the audio signal from a time-varying signal to a short-time stationary signal that can be analyzed in the frequency domain. Finally, feature extraction is performed on the audio signal after windowing each audio frame to obtain the audio in the audio and video sample data, i.e., speech data. The video processing library can be an application such as moviepy, the audio processing library can be an application such as librosa, and the window function can be a function such as a Hamming window.

[0088] It should be noted that in order to improve the accuracy of speech data extracted from audio and video sample data, after obtaining the audio signal, a series of audio signal preprocessing work is required to reduce the impact of noise in the audio signal. Among them, the audio signal preprocessing work can include noise reduction, dereverberation, gain control and other operations.

[0089] In step S103 of some embodiments, feature extraction methods such as Mel-frequency cepstral coefficients or linear prediction coding can be used to extract features from speech data, thereby obtaining key information of the speech data, namely, audio features. Secondly, the audio features are matched with known speech patterns using an acoustic model to obtain a target speech pattern of the audio features. Then, based on the target speech pattern, the most likely voice command or word corresponding to the audio feature is determined. Finally, a language model can be used to perform context recognition on the voice commands or words to obtain context information. Based on the context information, the voice commands or words are integrated to obtain a text representation of the speech data, namely, speech text, wherein the acoustic model can be a traditional method based on a hidden Markov model or a modern deep learning model.

[0090] In step S104 of some embodiments, video information corresponding to the voice data is extracted from the audio and video sample data to obtain video sample data, and then multimodal attribute extraction is performed on the audio and video sample data based on the voice data and the video sample data to obtain first attribute information of the voice data and second attribute information of the video sample data. Furthermore, the first attribute information and the second attribute information are attribute merged to obtain actual attribute information of the voice data, that is, audio attributes.

[0091] For details, see Figure 2 In some embodiments, step S104 may include but is not limited to steps S201 to S205:

[0092] Step S201, based on the voice data, performing video extraction on the audio and video sample data to obtain video sample data;

[0093] Step S202, extracting attributes from the speech data based on a preset audio attribute extraction model to obtain a first audio attribute and a first attribute confidence;

[0094] Step S203, performing face recognition on the video sample data to obtain a face image;

[0095] Step S204, based on a preset face attribute extraction model and a face image, attribute recognition is performed on the voice data to obtain a second audio attribute and a second attribute confidence;

[0096] Step S205: Generate an audio attribute based on the first audio attribute, the first attribute confidence, the second audio attribute and the second attribute confidence.

[0097] In step S201 of some embodiments, the voice playback time is obtained by locating the playback time of the voice data in the audio and video sample data, and video extraction is performed on the audio and video sample data according to the voice playback time, so as to obtain the video sample data corresponding to the voice data.

[0098] In step S202 of some embodiments, the first attribute confidence is used to represent the credibility of each attribute in the first audio attribute. For example, the first audio attribute includes a gender attribute, a speaking rate attribute, and an emotional attribute, wherein the credibility of the gender attribute is 80%, the credibility of the speaking rate attribute is 90%, and the credibility of the emotional attribute is 60%. Then the first attribute confidence is {0.8, 0.9, 0.6}.

[0099] The present application can parse information such as speaker gender, speaker emotion, and speaker speaking speed of the speech data according to speech features in the speech data using a pre-trained audio attribute extraction model, thereby obtaining a first audio attribute, and in parsing the speech features, calculate the credibility of information such as speaker gender, speaker emotion, and speaker speaking speed, thereby obtaining a first attribute confidence.

[0100] For details, see Figure 3 In some embodiments, step S202 may include but is not limited to steps S301 to S306:

[0101] Step S301, extracting features from the speech data to obtain audio fundamental frequency, Mel-spectrum cepstral coefficients, pitch features, and volume features;

[0102] Step S302, based on the audio attribute extraction model and the audio fundamental frequency, the speaker gender recognition is performed on the speech data to obtain the speaker gender attribute and the audio gender attribute confidence of the speaker gender attribute;

[0103] Step S303, based on the audio attribute extraction model, the Mel-spectrum cepstral coefficients and the pitch features, the emotion feature extraction is performed on the speech data to obtain the audio emotion attribute and the audio emotion attribute confidence of the audio emotion attribute;

[0104] Step S304, based on the audio attribute extraction model and the pitch feature and the volume feature, extract the speech rate feature of the speech data to obtain the audio speech rate attribute and the audio speech rate attribute confidence of the audio speech rate attribute;

[0105] Step S305, merging the speaker gender attribute, the audio emotion attribute and the audio speech speed attribute to obtain a first audio attribute;

[0106] Step S306: The audio gender attribute confidence, the audio emotion attribute confidence, and the audio speech rate attribute confidence are aggregated to obtain a first attribute confidence.

[0107] In step S301 of some embodiments, the audio fundamental frequency is used to reflect the pitch of the speaker's voice. The Mel-spectrogram cepstral coefficients are used to represent the frequency content of the speaker's voice. The pitch feature is used to reflect the melody and rhythm of the speaker's voice. The volume feature is used to reflect the loudness of the speaker's voice.

[0108] The present application can extract audio features such as audio fundamental frequency, Mel-spectrum cepstral coefficients, pitch features, and volume features from speech data through algorithms such as Fourier transform.

[0109] In step S302 of some embodiments, the audio attribute extraction model may be a machine learning model or a deep learning model, wherein the machine learning model may be a support vector machine and the deep learning model may be a neural network model.

[0110] The present application can utilize a trained audio attribute extraction model to analyze the input audio fundamental frequency, thereby calculating the probability that the gender of the speaker in the speech data is male, and the probability that the gender of the speaker is female. Furthermore, based on the above male probability and female probability, the possible gender of the speaker in the speech data is predicted to obtain the speaker's gender attribute, and then based on the speaker's gender attribute, male probability and female probability, the audio gender attribute confidence is generated.

[0111] In step S303 of some embodiments, the trained audio attribute extraction model can be used to perform feature analysis on the input Mel-spectrogram cepstral coefficients and pitch features, so as to predict the speaker's emotion in the speech data, that is, the audio emotion attribute, and generate the audio emotion attribute confidence of the audio emotion attribute.

[0112] In step S304 of some embodiments, the trained audio attribute extraction model can be used to perform feature analysis on the input pitch features and volume features, so as to predict the speaking speed of the speaker in the speech data, that is, the audio speaking speed attribute, and generate the audio speaking speed attribute confidence of the audio speaking speed attribute.

[0113] It should be noted that there are multiple modules inside the above audio attribute extraction model, which can generate corresponding attribute information and attribute confidence based on the input audio features.

[0114] In steps S305 and S306 of some embodiments, the first audio attribute can be obtained by combining the above-mentioned speaker gender attribute, audio emotion attribute and audio speaking speed attribute, and the first attribute confidence can be obtained by combining the above-mentioned audio gender attribute confidence, audio emotion attribute confidence and audio speaking speed attribute confidence.

[0115] In steps S301 to S306 shown in the present embodiment, by performing feature extraction on the speech data, the audio fundamental frequency, Mel-spectrogram cepstral coefficients, pitch features and volume features are obtained, and then the audio fundamental frequency, Mel-spectrogram cepstral coefficients, pitch features and volume features are feature analyzed according to the audio attribute extraction model. The key attributes such as the speaker's gender, emotion and speaking rate and the corresponding attribute confidence can be accurately extracted from the speech data. Furthermore, according to the attribute confidence of the key attributes such as gender, emotion and speaking rate, the accuracy of audio attribute recognition can be improved.

[0116] In step S203 of some embodiments, the video sample data is frame-processed to obtain video frames, face detection is performed on each video frame to obtain a face area of ​​the video frame, and then face alignment is performed on the face in the face area to obtain an aligned face video frame. Furthermore, the face video frames are merged and saved to obtain a face image.

[0117] In step S204 of some embodiments, a pre-trained facial attribute extraction model can be used to analyze the facial image to obtain information such as gender, emotion, and age of the facial image, that is, the second audio attribute, and in the process of analyzing the facial image, the credibility of the gender, emotion, and other information is calculated to obtain the second attribute confidence.

[0118] For details, see Figure 4 In some embodiments, step S204 may include but is not limited to steps S401 to S405:

[0119] Step S401, performing gender recognition on a face image based on a face attribute extraction model to obtain a face gender attribute and a face gender attribute confidence level of the face gender attribute;

[0120] Step S402, performing emotion recognition on the face image based on the face attribute extraction model to obtain the face emotion attribute and the face emotion attribute confidence of the face emotion attribute;

[0121] Step S403, predicting the age of the face image based on the face attribute extraction model, and obtaining the face age attribute and the face age attribute confidence of the face age attribute;

[0122] Step S404, merging the face gender attribute, the face emotion attribute and the face age attribute to obtain a second audio attribute;

[0123] Step S405, combining the face gender attribute confidence, the face emotion attribute confidence and the face age attribute confidence to obtain a second attribute confidence.

[0124] In steps S401 to S403 of some embodiments, the facial attribute extraction model may be a convolutional neural network-based model, such as VGG-Face (Very Deep Network for Face Recognition), OpenFace (Open-source Face Recognition with Deep Learning), and other models.

[0125] The facial attribute extraction model can be used to extract features from facial images to obtain facial features, and then based on the facial features, the gender information, emotional information and age information represented by the facial image can be identified to obtain facial gender attributes, facial emotional attributes and facial age attributes, and the probabilities of the above facial gender attributes, facial emotional attributes and facial age attributes are used as attribute confidences to obtain facial gender attribute confidence, facial emotional attribute confidence and facial age attribute confidence.

[0126] In step S404 and step S405 of some embodiments, a second audio attribute can be obtained by combining the above-mentioned face gender attribute, face emotion attribute and face age attribute, and a second attribute confidence can be obtained by combining the above-mentioned face gender attribute confidence, face emotion attribute confidence and face age attribute confidence.

[0127] In steps S401 to S405 shown in this embodiment, feature extraction and attribute recognition are performed on the facial image according to the facial attribute extraction model, and key attributes such as the speaker's gender, emotion, and age and the corresponding attribute confidence can be accurately extracted from the facial image. Furthermore, according to the attribute confidence of the above-mentioned key attributes such as gender, emotion, and age, the accuracy of attribute recognition of the facial image can be improved.

[0128] In step S205 of some embodiments, the confidence of the same attribute in the first audio attribute and the second audio attribute is compared to determine the trade-off of the same attribute to obtain the predicted audio attribute, and then the predicted audio attribute is merged with the different attributes in the first audio attribute and the second audio attribute to obtain the audio attribute.

[0129] For details, see Figure 5In some embodiments, step S205 may include but is not limited to steps S501 to S503:

[0130] Step S501, comparing the first audio attribute with the second audio attribute to obtain the same audio attribute and the different audio attribute, wherein the same audio attribute refers to the same attribute as the second audio attribute in the first audio attribute, and the different audio attribute refers to the different attribute between the first audio attribute and the audio attribute;

[0131] Step S502, based on the first attribute confidence and the second attribute confidence, attribute selection is performed on the same audio attribute to obtain a predicted audio attribute;

[0132] Step S503: Merge the predicted audio attribute with different audio attributes to obtain audio attributes.

[0133] In step S501 of some embodiments, by comparing the first audio attribute and the second audio attribute, the same attributes and different attributes existing in the two can be obtained, that is, the same audio attributes and different audio attributes. For example, the first audio attribute includes gender attributes, emotion attributes and speaking speed attributes, and the second audio attributes include gender attributes, emotion attributes and age attributes. Then the same audio attributes are gender attributes and emotion attributes, and the different audio attributes are speaking speed attributes and age attributes.

[0134] In step S502 of some embodiments, the first attribute confidence and the second attribute confidence of the same audio attribute are compared in confidence, and the audio attribute with the larger confidence between the first attribute confidence and the second attribute confidence is selected as the attribute of the same audio attribute to obtain the predicted audio attribute. For example, when the same audio attribute is a gender attribute, the first audio attribute indicates that the gender attribute is male, and the first attribute confidence indicates that the confidence of the gender attribute is 0.8, and the second audio attribute indicates that the gender attribute is female, and the second attribute confidence indicates that the confidence of the gender attribute is 0.6, then the attribute selection of the gender attribute is male, and the gender attribute representation of male is used as the predicted audio attribute.

[0135] In step S503 of some embodiments, the attribute selection of the predicted audio attribute is combined with the attribute selection of the different audio attribute to form the actual attribute of the speech data, that is, the audio attribute.

[0136] In steps S501 to S503 shown in this embodiment, the first audio attribute and the second audio attribute are compared to obtain the same audio attribute and the different audio attributes, and then the same audio attribute is selected according to the first attribute confidence and the second attribute confidence to obtain the predicted audio attribute. Furthermore, the predicted audio attribute and the different audio attribute are merged to obtain the audio attribute, which can improve the accuracy of audio attribute recognition and extraction.

[0137] In steps S201 to S205 shown in this embodiment, video extraction is performed on the audio and video sample data according to the voice data to obtain video sample data, and then attribute extraction is performed on the voice data according to the preset audio attribute extraction model to obtain the first audio attribute and the first attribute confidence. Further, face recognition is performed on the video sample data to obtain a face image. Then, attribute recognition is performed on the voice data according to the preset face attribute extraction model and the face image to obtain the second audio attribute and the second attribute confidence. Finally, audio attributes are generated according to the first audio attribute, the first attribute confidence, the second audio attribute and the second attribute confidence, so as to achieve the extraction and recognition of rich attributes in the voice data from multiple dimensions, thereby obtaining a comprehensive and accurate audio attribute description, thereby improving the accuracy and depth of voice data attribute recognition.

[0138] In step S105 of some embodiments, by filling the audio attribute into the attribute description template, a single attribute description text for describing one of the audio attributes of the audio and video sample data can be obtained.

[0139] For details, see Figure 6 In some embodiments, step S105 may include but is not limited to steps S601 to S602:

[0140] Step S601, constructing an attribute description template based on audio and video sample data;

[0141] Step S602: Fill in the attribute description template based on the audio attribute to obtain a single attribute description text.

[0142] In step S601 of some embodiments, the attribute description template refers to a predefined text frame used to carry the audio attributes that are subsequently filled in.

[0143] This application can determine the template structure and attribute description method based on the audio and video sample data, and create attribute fields based on the template structure and attribute description method for inserting specific audio attributes to obtain an attribute description template.

[0144] In step S602 of some embodiments, different attribute description templates are selected according to different audio attributes, and then the audio attributes are inserted into corresponding positions in the attribute description templates to obtain a single attribute description text.

[0145] In steps S601 to S602 shown in this embodiment, an attribute description template is constructed based on the audio and video sample data, and then the attribute description template is filled in according to the audio attributes to obtain a single attribute description text, so as to achieve a natural language description of the audio attributes, thereby facilitating the subsequent construction of the speech synthesis instruction text.

[0146] In step S106 of some embodiments, a speech synthesis instruction text can be obtained by merging multiple single-attribute description texts, and then the speech synthesis instruction text is concatenated with the speech text to obtain speech synthesis sample data for training a speech synthesis model.

[0147] For details, see Figure 7 In some embodiments, step S106 may include but is not limited to steps S701 to S702:

[0148] Step S701, merging multiple single-attribute description texts to obtain speech synthesis instruction text;

[0149] Step S702: concatenate the speech synthesis instruction text with the speech text to obtain speech synthesis sample data.

[0150] In step S701 of some embodiments, the above-mentioned multiple single-attribute description texts describing different audio attributes are sorted and merged to obtain a speech synthesis instruction text. For example, there are single-attribute description texts, namely text a: {the speaker's voice shows male characteristics}, text b: {the speaker's voice shows female characteristics}, text c: {the voice belongs to a 16-year-old speaker}, text d: {the voice belongs to a 30-year-old speaker}. By sorting and merging text a and text c, a speech synthesis instruction text can be obtained: {the speaker's voice shows the characteristics of a 16-year-old male} or {the voice belongs to a 16-year-old male speaker}. It should be noted that single-attribute description texts describing the same audio data cannot be merged. For example, the above-mentioned text a and the above-mentioned text b cannot be merged, and the above-mentioned text c and the above-mentioned text d cannot be merged.

[0151] In step S702 of some embodiments, the speech synthesis instruction text and the speech text are concatenated to obtain speech synthesis sample data for training a speech synthesis model.

[0152] In steps S701 to S702 shown in the present embodiment, a plurality of single-attribute description texts are merged to obtain speech synthesis instruction text, and then the speech synthesis instruction text and the speech text are concatenated to obtain speech synthesis sample data, which can effectively increase the amount of speech synthesis sample data, thereby improving the accuracy of speech synthesis model training.

[0153] In step S107 of some embodiments, the speech synthesis sample data is input into a pre-built speech synthesis model for unsupervised model learning, thereby achieving model training for the speech synthesis model and improving the accuracy of the speech synthesis model training.

[0154] The present application obtains rich audio and video sample data and performs audio attribute recognition on the voice data to obtain audio attributes, and then describes the audio attributes of the audio and video sample data based on multiple audio attributes to obtain multiple single attribute description texts, thereby solving the problem of scarce instruction text. Subsequently, voice extraction is performed on the audio and video sample data to obtain voice data, and then audio text recognition is performed on the voice data to obtain voice text. Then, based on the voice text, the voice synthesis instruction text is instantiated to obtain voice synthesis sample data, and based on the voice synthesis sample data, a preset voice synthesis model is trained to ensure that the voice synthesis model receives more comprehensive sample data, thereby improving the efficiency and accuracy of the voice synthesis model training and optimizing the quality of voice synthesis.

[0155] See also Figure 8 The present application also provides a speech synthesis model training device, which can implement the above-mentioned speech synthesis model training method, and the device includes:

[0156] The data acquisition module 801 is used to acquire audio and video sample data;

[0157] The speech recognition module 802 is used to extract speech from the audio and video sample data to obtain speech data;

[0158] The audio recognition module 803 is used to perform audio text recognition on the voice data to obtain the voice text;

[0159] The attribute recognition module 804 is used to perform audio attribute recognition on the speech data to obtain audio attributes;

[0160] The attribute description module 805 is used to perform audio attribute description on the audio and video sample data based on multiple audio attributes to obtain multiple single attribute description texts, wherein the single attribute description text is used to describe one of the audio attributes of the audio and video sample data;

[0161] The text instantiation module 806 is used to instantiate the single attribute description text based on the speech text to obtain speech synthesis sample data;

[0162] The model training module 807 is used to perform model training on a preset speech synthesis model based on speech synthesis sample data.

[0163] The specific implementation of the speech synthesis model training device is basically the same as the specific implementation of the above-mentioned speech synthesis model training method, and will not be repeated here.

[0164] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned speech synthesis model training method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0165] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0166] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0167] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the speech synthesis model training method of the embodiment of this application;

[0168] Input / output interface 903, used to implement information input and output;

[0169] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0170] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0171] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0172] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned speech synthesis model training method is implemented.

[0173] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0174] The speech synthesis model training method, speech synthesis model training device, electronic device and storage medium provided in the embodiments of the present application perform speech extraction on the acquired audio and video sample data to obtain speech data, then perform audio text recognition on the speech data to obtain speech text, then perform audio attribute recognition on the audio and video sample data to obtain audio attributes, then perform audio attribute description on the audio and video sample data based on multiple audio attributes to obtain multiple single attribute description texts for describing one of the audio attributes of the audio and video sample data, further, perform instantiation processing on the single attribute description text based on the speech text to obtain speech synthesis sample data, and finally, perform model training on a preset speech synthesis model based on the speech synthesis sample data.

[0175] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0176] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0177] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0179] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0180] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0181] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0182] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0184] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0185] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A speech synthesis model training method, characterized in that: The method comprises: Get audio and video sample data; Performing voice extraction on the audio and video sample data to obtain voice data; Performing audio-to-text recognition on the voice data to obtain voice text; Performing audio attribute recognition on the voice data to obtain audio attributes; Based on the multiple audio attributes, the audio and video sample data is described in audio attributes to obtain multiple single attribute description texts, wherein the single attribute description text is used to describe one of the audio attributes of the audio and video sample data; Based on the speech text, instantiate the single attribute description text to obtain the speech synthesis sample data; Based on the speech synthesis sample data, model training is performed on a preset speech synthesis model.

2. The method according to claim 1, characterized in that: The step of performing audio attribute recognition on the voice data to obtain audio attributes includes: Based on the voice data, performing video extraction on the audio and video sample data to obtain video sample data; Based on a preset audio attribute extraction model, extract attributes from the speech data to obtain a first audio attribute and a first attribute confidence; Performing face recognition on the video sample data to obtain a face image; Based on a preset face attribute extraction model and the face image, perform attribute recognition on the voice data to obtain a second audio attribute and a second attribute confidence; The audio attribute is generated based on the first audio attribute, the first attribute confidence, the second audio attribute, and the second attribute confidence.

3. The method according to claim 2, characterized in that The method of extracting attributes from the speech data based on a preset audio attribute extraction model to obtain a first audio attribute and a first attribute confidence level includes: Extracting features from the speech data to obtain audio fundamental frequency, Mel-spectrum cepstral coefficients, pitch features, and volume features; Based on the audio attribute extraction model and the audio fundamental frequency, the speaker gender recognition is performed on the speech data to obtain the speaker gender attribute and the audio gender attribute confidence of the speaker gender attribute; Based on the audio attribute extraction model, the Mel-spectrogram cepstral coefficients and the pitch features, extracting emotional features from the speech data to obtain audio emotional attributes and audio emotional attribute confidences of the audio emotional attributes; Based on the audio attribute extraction model, the tone feature, and the volume feature, extracting speech rate features from the speech data to obtain an audio speech rate attribute and an audio speech rate attribute confidence of the audio speech rate attribute; Merging the speaker gender attribute, the audio emotion attribute, and the audio speech speed attribute to obtain the first audio attribute; The audio gender attribute confidence, the audio emotion attribute confidence, and the audio speaking speed attribute confidence are aggregated to obtain the first attribute confidence.

4. The method according to claim 2, characterized in that: The step of performing attribute recognition on the voice data based on the preset face attribute extraction model and the face image to obtain a second audio attribute and a second attribute confidence level includes: Performing gender recognition on the face image based on the face attribute extraction model to obtain a face gender attribute and a face gender attribute confidence level of the face gender attribute; Performing emotion recognition on the face image based on the face attribute extraction model to obtain face emotion attributes and face emotion attribute confidence levels of the face emotion attributes; Performing age prediction on the face image based on the face attribute extraction model to obtain a face age attribute and a face age attribute confidence level of the face age attribute; Merging the face gender attribute, the face emotion attribute and the face age attribute to obtain the second audio attribute; The face gender attribute confidence, the face emotion attribute confidence and the face age attribute confidence are combined to obtain the second attribute confidence.

5. The method according to claim 2, characterized in that: The step of generating the audio attribute based on the first audio attribute, the first attribute confidence, the second audio attribute and the second attribute confidence includes: Comparing the first audio attribute with the second audio attribute to obtain the same audio attribute and the different audio attribute, wherein the same audio attribute refers to the same attribute as the second audio attribute in the first audio attribute, and the different audio attribute refers to the different attribute between the first audio attribute and the audio attribute; Based on the first attribute confidence and the second attribute confidence, performing attribute selection on the same audio attribute to obtain a predicted audio attribute; The predicted audio attribute and the different audio attribute are merged to obtain the audio attribute.

6. The method according to any one of claims 1 to 5, characterized in that: The step of performing audio attribute description on the audio and video sample data based on the multiple audio attributes to obtain multiple single attribute description texts includes: Based on the audio and video sample data, construct an attribute description template; Based on the audio attribute, the attribute description template is filled to obtain the single attribute description text.

7. The method according to any one of claims 1 to 5, characterized in that: The instantiation processing of the single attribute description text based on the speech text to obtain the speech synthesis sample data includes: Merging the plurality of single-attribute description texts to obtain a speech synthesis instruction text; The speech synthesis instruction text is concatenated with the speech text to obtain the speech synthesis sample data.

8. A speech synthesis model training device, characterized in that: The device comprises: A data acquisition module, used to acquire audio and video sample data; A speech recognition module, used to perform speech extraction on the audio and video sample data to obtain speech data; An audio recognition module, used to perform audio text recognition on the voice data to obtain voice text; An attribute recognition module, used to perform audio attribute recognition on the voice data to obtain audio attributes; An attribute description module, used for performing audio attribute description on the audio and video sample data based on the multiple audio attributes to obtain multiple single attribute description texts, wherein the single attribute description text is used to describe one of the audio attributes of the audio and video sample data; A text instantiation module, used for instantiating the single attribute description text based on the speech text to obtain the speech synthesis sample data; The model training module is used to perform model training on a preset speech synthesis model based on the speech synthesis sample data.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the speech synthesis model training method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech synthesis model training method described in any one of claims 1 to 7 is implemented.

Images