Voice generation method and device, electronic equipment and storage medium

Abstract

The invention discloses a voice generation method and apparatus, an electronic device and a readable storage medium. The method comprises the steps of obtaining a target text for generating a target voice and target scene data; the target scene data is used for indicating a playing scene targeted by the target voice; processing based on the target scene data through a mood classification model, a speech speed regression model and a volume regression model to obtain a target mood predicted by the mood classification model, a target speech speed predicted by the speech speed regression model and a target volume predicted by the volume regression model; and based on the target tone, the target speech speed and the target volume, performing speech generation on the target text to obtain target speech. According to the method provided by the invention, the timbre of the generated target voice is diversified, the situation that the timbre of the voice is single due to the fact that the voice is generated by adopting fixed voice parameters is avoided, the generated target voice can meet different playing scenes, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of software testing technology, and more specifically, to a speech generation method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Speech generation technology involves generating speech from text based on given speech parameters, ensuring that the generated speech conforms to the given speech parameters. However, existing methods often generate monotonous speech, making it difficult to meet diverse user needs and resulting in a poor user experience. Summary of the Invention

[0003] This application proposes a speech generation method, device, electronic device and computer-readable storage medium to improve the above-mentioned defects.

[0004] In a first aspect, an embodiment of the present application provides a method for generating speech, the method comprising:

[0005] Obtaining target text and target scene data for generating a target speech; the target scene data is used to indicate a playback scene targeted by the target speech;

[0006] The target scene data is processed by the tone classification model, the speaking rate regression model, and the volume regression model respectively to obtain the target tone predicted by the tone classification model, the target speaking rate predicted by the speaking rate regression model, and the target volume predicted by the volume regression model; wherein the tone classification model, the speaking rate regression model, and the volume regression model are trained based on the target loss, and the target loss is determined based on the first loss, the second loss, and the third loss; the first loss indicates the accuracy of the classification prediction of the tone classification model, the second loss indicates the accuracy of the regression prediction of the speaking rate regression model, and the third loss indicates the accuracy of the regression prediction of the volume regression model;

[0007] Based on the target tone, target speaking speed and target volume, speech generation is performed on the target text to obtain the target speech.

[0008] In a second aspect, an embodiment of the present application further provides a speech generation device, the device comprising:

[0009] An acquisition module is used to acquire target text and target scene data for generating a target speech; the target scene data is used to indicate a playback scene targeted by the target speech;

[0010] a data processing module for processing target scene data using a tone classification model, a speech rate regression model, and a volume regression model, respectively, to obtain a target tone predicted by the tone classification model, a target speech rate predicted by the speech rate regression model, and a target volume predicted by the volume regression model; wherein the tone classification model, the speech rate regression model, and the volume regression model are trained based on a target loss, and the target loss is determined based on a first loss, a second loss, and a third loss; the first loss indicates the accuracy of the classification prediction of the tone classification model, the second loss indicates the accuracy of the regression prediction of the speech rate regression model, and the third loss indicates the accuracy of the regression prediction of the volume regression model;

[0011] The speech generation module is used to generate speech from the target text based on the target tone, target speaking speed and target volume to obtain the target speech.

[0012] In a third aspect, an embodiment of the present application further provides an electronic device, characterized in that the electronic device includes: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by one or more processors, and the one or more programs are configured to execute the above method.

[0013] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program code executable by a processor, and when the program code is executed by the processor, the processor executes the above method.

[0014] The present application provides a speech generation method, device, electronic device and computer-readable storage medium. In the present application, based on target scene data indicating the playback scene targeted by the target speech, the target tone, target speech speed and target volume are determined, and then based on the target tone, target speech speed and target volume, speech generation is performed on the target text to obtain the target speech. Thus, the purpose of determining the three speech parameters of target tone, target speech speed and target volume according to the target scene data is achieved, and the purpose of determining different speech parameters for different playback scenes is achieved, so that the timbre of the generated target speech is diversified, avoiding the situation where the speech timbre is single due to the use of fixed speech parameters to generate speech. The generated target speech can meet different playback scenarios and improve the user experience. At the same time, the tone classification model, speech rate regression model and volume regression model are trained based on the target loss, and the target loss is determined based on the first loss of the tone classification model, the second loss of the speech rate regression model and the third loss of the volume regression model, so that the target loss integrates the prediction accuracy of the tone classification model, the speech rate regression model and the volume regression model, and realizes the fusion training of the tone classification model, the speech rate regression model and the volume regression model. The training processes of the tone classification model, the speech rate regression model and the volume regression model are interrelated, which improves the prediction ability of the trained tone classification model, the speech rate regression model and the volume regression model, thereby improving the accuracy of the target tone, target speech rate and target volume predicted by the tone classification model, the speech rate regression model and the volume regression model respectively, so that the target speech generated based on the target tone, target speech rate and target volume is more in line with the targeted playback scenario, thereby improving the speech generation effect.

[0015] Other features and advantages of the embodiments of the present application will be described in the following description and, in part, will become apparent from the description or be understood by practicing the embodiments of the present application. The objectives and other advantages of the embodiments of the present application can be achieved and obtained through the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A flow chart of a speech generation method proposed according to an embodiment of the present application is shown.

[0018] Figure 2 Shown Figure 1The corresponding step S120 of the embodiment is a flowchart in one embodiment.

[0019] Figure 3 A schematic diagram showing a process of determining speech parameters of a target speech in an embodiment of the present application is shown.

[0020] Figure 4 The figure shows a structural block diagram of a speech generation device proposed in one embodiment of the present application.

[0021] Figure 5 A structural block diagram of an electronic device provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work fall within the scope of protection of the present application.

[0023] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0024] See also Figure 1 , Figure 1 A flow chart of a speech generation method proposed according to one embodiment of the present application is shown, which is used in an electronic device. The method includes:

[0025] S110: Acquire target text and target scene data for generating target speech.

[0026] The executor of the speech generation method of the present application is an electronic device, which may be an aircraft, a vehicle, a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device, an intelligent service terminal, etc.

[0027] The scenario data indicates the playback scenario for the generated speech. The playback scenario refers to a collection of parameters such as the playback environment of the device playing the speech, the user in the playback environment, the user's speech, and the contextual information of the text used to generate the speech. For example, the scenario data may include the user identifier, age, and gender of the user in the playback environment of the generated speech, an environmental description of the playback environment, information about the dialogue scene in the playback environment, and the contextual information of the text used to generate the speech.

[0028] In other words, the target scenario data is used to indicate the playback scenario for the target voice. For example, the target scenario data includes the target user identifier, age, gender, and environment description of the target playback environment within the target voice's target playback environment, dialogue scenario information within the target playback environment, and contextual information about the target text. The target playback environment refers to the playback environment of the target voice.

[0029] The playback environment may be the environment where the electronic device is located. For example, if the electronic device is a vehicle, the playback environment is inside the vehicle. For another example, if the electronic device is a mobile terminal, the playback environment is within 6 meters of the mobile terminal.

[0030] The target user refers to a user in the playback environment, and the identifier used to indicate the user is used as the user identifier, that is, the target user identifier is the user identifier of the target user, wherein the user identifier can be in the form of a combination of numbers or letters, such as the user identifier 0001 or 0022.

[0031] The environmental description information is used to describe the environment in which the electronic device is located. The form of the environmental description information is related to the type of the electronic device. The form of the environmental description information may be different for different types of electronic devices. For example, if the electronic device is a mobile terminal, the environmental description information may refer to the environmental category of the environment in which the electronic device is located. The environmental category may be a shopping mall, a cinema, a playground, or a park, etc. For another example, if the electronic device is a vehicle, the environmental description information may be the road section type of the road section on which the vehicle is traveling. The road section type may be a highway, an urban road, or a rural road, etc.

[0032] The conversation scene information of the playback environment refers to information used to describe the conversation situation of users in the playback environment. The conversation scene information can be, for example, a chatting scene, a music listening scene, and a news listening scene.

[0033] The contextual information of a text refers to the text content before and after the text. For example, if the text is a description of today's weather, the contextual information can be a description of yesterday's weather and tomorrow's weather. For another example, if the text is information related to road conditions, the contextual information can be clothing suggestions and transportation suggestions based on the weather.

[0034] In this embodiment, the ambient voice and image data of the playback scene targeted by the target voice are collected, and then the collected ambient voice and image data are analyzed by combining voice recognition and image recognition technologies to determine the aforementioned target scene data. The voice recognition model for recognizing ambient voice and the image recognition model for recognizing ambient image data can be a basic neural network model or a large language model (LLM), which is not limited in this application.

[0035] S120, processing is performed based on the target scene data through the tone classification model, the speech rate regression model and the volume regression model respectively, to obtain the target tone predicted by the tone classification model, the target speech rate predicted by the speech rate regression model and the target volume predicted by the volume regression model.

[0036] Among them, the tone classification model, speech rate regression model and volume regression model are trained based on the target loss, and the target loss is determined based on the first loss, the second loss and the third loss; the first loss indicates the accuracy of the classification prediction of the tone classification model, the second loss indicates the accuracy of the regression prediction of the speech rate regression model, and the third loss indicates the accuracy of the regression prediction of the volume regression model.

[0037] Generally speaking, tone is a discrete value, so the tone classification model can be a decision tree model for classification decision, which is used to make classification decisions on discrete tones and obtain the target tone; speaking rate is a continuous value, so the speaking rate regression model can be a decision tree model for regression decision, which is used to perform regression modeling and predict the target speaking rate within a continuous speaking rate range; volume is also a continuous value, so the volume regression model can be a decision tree model for regression decision, which is used to perform regression modeling and predict the target volume within a continuous volume range.

[0038] In this application, the tone involved may include gentle, angry, calm, ordinary, etc., and the speech rate range involved is 0-200%, wherein the speech rate range is a proportional parameter based on a preset speech rate, and the preset speech rate is, for example, 1 word per second. The volume range involved is 0-200%, and the volume range is a proportional parameter based on the maximum volume of the electronic device.

[0039] In this application, the relevant parameters in each decision tree model involved are as follows:

[0040] class_weight: Set the weight of each class sample. The default is that the weight of each sample is the same, all are 1;

[0041] ccp_alpha: alpha coefficient during pruning. Set this parameter when pruning is required.

[0042] random_state: This parameter needs to be set if the training should be the same each time;

[0043] criterion: node quality evaluation function (gini, entropy).

[0044] In order to prevent overfitting in the training process of the decision tree model, the following parameters in the decision tree model can also be adaptively adjusted:

[0045] min_samples_leaf: minimum number of samples for leaf nodes;

[0046] min_samples_split: minimum number of samples for node branching;

[0047] max_depth: the maximum depth of the tree branch;

[0048] min_weight_fraction_leaf: minimum weight sum of leaf nodes;

[0049] min_impurity_decrease: minimum purity increase of node branches

[0050] max_leaf_nodes: Maximum number of leaf nodes.

[0051] Diversity parameters for decision tree models can include:

[0052] max_features: the maximum number of features to search;

[0053] splitter: variable selection method during branching.

[0054] The tone classification model, speech rate regression model and volume regression model are obtained by adjusting parameters simultaneously through target loss. The target loss is jointly determined by the first loss, second loss and third loss corresponding to the tone classification model, speech rate regression model and volume regression model respectively, thereby integrating the losses of the three models for fusion training, realizing the mutual correlation between the training processes of the tone classification model, speech rate regression model and volume regression model, and improving the model capabilities of the tone classification model, speech rate regression model and volume regression model.

[0055] In some embodiments, the training process of the tone classification model, speech rate regression model, and volume regression model may include: obtaining sample scene data and sample annotation data corresponding to the sample speech; the sample annotation data includes the sample tone, sample speech rate, and sample volume of the sample speech; processing the sample scene data by the tone classification model, the speech rate regression model, and the volume regression model, respectively, to obtain a predicted sample tone predicted by the tone classification model, a predicted sample speech rate predicted by the speech rate regression model, and a predicted sample volume predicted by the volume regression model; determining a first loss based on the predicted sample tone and the sample tone; determining a second loss based on the predicted sample speech rate and the sample speech rate; determining a third loss based on the predicted sample volume and the sample volume; determining a target loss based on the first loss, the second loss, and the third loss; and training the tone classification model, the speech rate regression model, and the volume regression model using the target loss until a training termination condition is met. The training termination condition may be that the target loss is less than a loss threshold or the number of iterations reaches a maximum number of iterations. The loss threshold and the maximum number of iterations may be set based on requirements, such as the loss threshold being 0.1 and the maximum number of iterations being 1000.

[0056] Among them, the sample voice can be a voice with high user satisfaction under the constraint of the number of sample scene data (or an ideal voice under a relatively ideal state), the tone of the sample voice is the sample tone, the speaking speed of the sample voice is the sample speaking speed, and the volume of the sample voice is the sample volume.

[0057] In this case, the sample scenario data is used to indicate the playback scenario for the sample speech; the sample scenario data may include the sample user identifier, age, gender, and environment description of the sample playback environment within the sample speech's sample playback environment, as well as conversation scenario information within the sample playback environment. The sample user refers to the user within the sample playback environment, and the sample playback environment refers to the playback environment of the sample speech.

[0058] In some implementations, the sample speech may be generated based on sample text. In this case, the sample scenario data may also include contextual information about the sample text. In this case, the sample tone refers to the tone of the sample speech generated when the sample text is generated under ideal conditions, subject to the constraints of the sample scenario data. Similarly, the sample speech rate refers to the speech rate of the sample speech generated when the sample text is generated under ideal conditions, subject to the constraints of the sample scenario data. Similarly, the sample volume refers to the volume of the sample speech generated when the sample text is generated under ideal conditions, subject to the constraints of the sample scenario data.

[0059] The method for determining the sample scene data is similar to the method for determining the target scene data, and will not be described in detail here. Alternatively, sample annotation data can be determined based on the sample scene data based on manual experience, and then speech generation can be performed based on the sample text based on the sample annotation data to obtain sample speech.

[0060] A parameter-initialized tone classification model, a parameter-initialized speech rate regression model, and a parameter-initialized volume regression model can be obtained, and then the parameter-initialized tone classification model, the parameter-initialized speech rate regression model, and the parameter-initialized volume regression model are used to process the sample scene data to obtain the predicted sample tone predicted by the tone classification model, the predicted sample speech rate predicted by the speech rate regression model, and the predicted sample volume predicted by the volume regression model.

[0061] Afterwards, the cross entropy loss function or the mean square error loss function can be calculated based on the difference between the predicted sample tone and the sample tone to obtain the first loss, and the regularized maximum likelihood function can be calculated based on the predicted sample speech rate and the sample speech rate to obtain the second loss, and the regularized maximum likelihood function can be calculated based on the predicted sample volume and the sample volume to obtain the third loss.

[0062] Afterwards, the first loss, the second loss, and the third loss can be fused to obtain the target loss. This can be achieved by performing a weighted summation of the first loss, the second loss, and the third loss to obtain the target loss. The weights of the first loss, the second loss, and the third loss can be set based on requirements and are not limited here.

[0063] At this time, the calculation process of the target loss is simplified to: L = α1*L1+α2*L2+α3*L3, L is the target loss, L1 is the first loss, L2 is the second loss, L3 is the third loss, α1 is the weight of the first loss, α2 is the weight of the second loss, and α3 is the weight of the third loss.

[0064] S130 : Based on the target tone, target speaking speed, and target volume, generate speech for the target text to obtain a target speech.

[0065] After obtaining the target tone, target speaking speed, and target volume, speech generation is performed on the target text based on the target tone, target speaking speed, and target volume to obtain the target speech. The target speech has the target tone, the target volume, the target speaking speed, and the target text.

[0066] In this embodiment, the target speech can be generated by an existing speech generation model, and the speech generation model can be, for example, Doubao, chat-gpt, Wenxinyiyan, and Tongyiqianwen.

[0067] In this embodiment, based on the target scene data indicating the playback scene for the target voice, the target tone, target speech speed and target volume are determined, and then based on the target tone, target speech speed and target volume, voice generation is performed on the target text to obtain the target voice. Thus, the purpose of determining the three voice parameters of target tone, target speech speed and target volume according to the target scene data is achieved, and the purpose of determining different voice parameters for different playback scenes is achieved, so that the timbre of the generated target voice is diversified, avoiding the situation where the voice timbre is single due to the use of fixed voice parameters to generate voice. The generated target voice can meet different playback scenes and improve the user experience.

[0068] At the same time, the tone classification model, speech rate regression model and volume regression model are trained based on the target loss, and the target loss is determined based on the first loss of the tone classification model, the second loss of the speech rate regression model and the third loss of the volume regression model, so that the target loss integrates the prediction accuracy of the tone classification model, the speech rate regression model and the volume regression model, and realizes the fusion training of the tone classification model, the speech rate regression model and the volume regression model. The training processes of the tone classification model, the speech rate regression model and the volume regression model are interrelated, which improves the prediction ability of the trained tone classification model, the speech rate regression model and the volume regression model, thereby improving the accuracy of the target tone, target speech rate and target volume predicted by the tone classification model, the speech rate regression model and the volume regression model respectively, so that the target speech generated based on the target tone, target speech rate and target volume is more in line with the targeted playback scenario, thereby improving the speech generation effect.

[0069] In one embodiment, if Figure 2 As shown, S120 may include:

[0070] S210: Perform feature embedding processing on the target scene data through the large model to obtain target scene features.

[0071] The large model can be Doubao, Chat-GPT, Wenxinyiyan, or Tongyiqianwen. It can also be a BERT model. The large model performs feature embedding processing on the target scene data to effectively extract features. The result of feature embedding processing is used as the target scene feature.

[0072] Embedding a large model involves using a low-dimensional vector to represent an object, such as a word, a categorical feature (such as a product, movie, or item), or a time series feature. The embedding result more accurately represents the inherent meaning of the corresponding feature, ensuring that objects corresponding to vectors with similar geometric distances have similar meanings.

[0073] By embedding the target scene data into features using a large model, we can leverage the embedding capabilities of the large model to accurately extract features and obtain target scene features with higher accuracy.

[0074] S220. Perform classification prediction of the tone based on the target scene features using a tone classification model to obtain a target tone; perform regression prediction of the speech rate based on the target scene features using a speech rate regression model to obtain a target speech rate; perform regression prediction of the volume based on the target scene features using a volume regression model to obtain a target volume.

[0075] That is, the target scene features are respectively input into the tone classification model, the speech rate regression model and the volume regression model to obtain the target tone output by the tone classification model, the target speech rate output by the speech rate regression model and the target volume output by the volume regression model.

[0076] In this embodiment, the process of determining the speech parameters of the target speech (including the target tone, target speaking speed and target volume) is as follows: Figure 3 As shown in the figure, the target scene data is first input into the large model for feature embedding processing to obtain the target scene features. The target scene features are then input into the tone classification model for tone classification modeling to output the target tone. The target scene features are then input into the speech rate regression model for speech rate regression modeling to output the target speech rate. The target scene features are then input into the volume regression model for volume regression modeling to output the target volume.

[0077] It is worth mentioning that during the training process of the tone classification model, speech rate regression model and volume regression model, the sample scene data can be feature embedded through the large model to obtain the sample scene features, and then the sample scene features are input into the tone classification model, speech rate regression model and volume regression model respectively to obtain the predicted sample tone output by the tone classification model, the predicted sample speech rate output by the speech rate regression model and the predicted sample volume output by the volume regression model.

[0078] In this embodiment, feature embedding processing is performed on the scene data through a large model, so that effective scene features (including target scene features extracted from the target scene data and sample scene features extracted from the sample scene data) can be extracted from the scene data. Therefore, when the tone classification model, the speech rate regression model and the volume regression model make predictions based on the scene features, the prediction capabilities of the tone classification model, the speech rate regression model and the volume regression model can be brought into play to obtain prediction results with higher accuracy, thereby achieving accurate prediction of the target tone, target speech rate and target volume, and making the generated target speech more in line with the needs.

[0079] In one embodiment, the sample scene data includes a sample user identifier of a sample user in a sample playback environment of the sample voice; the target scene data includes a target user identifier of a target user in a target playback environment of the target voice. Accordingly, S120 includes: if there is a matching user identifier in the sample user identifier that matches the target user identifier, the target scene data is processed respectively by a tone classification model, a speech rate regression model, and a volume regression model to obtain a target tone predicted by the tone classification model, a target speech rate predicted by the speech rate regression model, and a target volume predicted by the volume regression model.

[0080] The presence of a matching user identifier in the sample user identifier that matches the target user identifier means that the sample users include the aforementioned target user, and the data of the target user exists in the data used to train the tone classification model, the speech rate regression model, and the volume regression model. The tone classification model, the speech rate regression model, and the volume regression model can accurately predict the voice parameters based on the target user identifier, age, and gender of the target user, and obtain voice parameters with higher accuracy (that is, the target tone, target speech rate, and target volume). At this time, the tone classification model, the speech rate regression model, and the volume regression model can be used to process the target scene data to obtain the target tone predicted by the tone classification model, the target speech rate predicted by the speech rate regression model, and the target volume predicted by the volume regression model.

[0081] It can be understood that when there is a matching user identifier in the sample user identifier that matches the target user identifier, the target scene data can also be processed by feature embedding through the large model to obtain the target scene features; then the tone classification model is used to classify and predict the tone based on the target scene features to obtain the target tone; the speech rate regression model is used to regress and predict the speech rate based on the target scene features to obtain the target speech rate; and the volume regression model is used to regress and predict the volume based on the target scene features to obtain the target volume.

[0082] For example, in the case where the electronic device is a vehicle, the scenario data includes the user ID, age, gender, the road type of the vehicle, and the conversation scenario information. For different scenario data, the determined tone, speaking speed, and volume are as shown in Table 1. Table 1 is as follows:

[0083] Table 1

[0084] User ID age gender road section Scenario volume speaking speed Tone 00001 young children female Urban roads Storytelling 70％ 75％ gentle 00022 middle aged female Expressway section Vehicle Control 65％ 100％ standard 00045 elderly male Rural roads Listen to the news 90％ 60％ patience

[0085] Among them, the scene in Table 1 refers to the dialogue scene information, and the road section refers to the road section type.

[0086] In this embodiment, there is a matching user identifier in the sample user identifier that matches the target user identifier, which means that the user identifier, age, gender and other data of the target user are involved in the training process of the tone classification model, the speech rate regression model and the volume regression model. The tone classification model, the speech rate regression model and the volume regression model can make accurate predictions based on the user identifier, age and gender and other data of the target user. It is necessary to combine the target scene data to predict the target tone, target volume and target speech rate. At this time, the target tone, target volume and target speech rate are predicted based on the target scene data by the tone classification model, the speech rate regression model and the volume regression model. The accuracy of the target tone, target volume and target speech rate is high, so that the generated target speech is more in line with the needs.

[0087] In one embodiment, the method further includes: if there is no matching user identifier in the sample user identifier that matches the target user identifier, obtaining a preset reference tone, reference speaking speed, and reference volume; and generating speech for the target text based on the reference tone, reference speaking speed, and reference volume to obtain a target speech.

[0088] The reference tone, reference speaking speed and reference volume can be set based on requirements. For example, the reference tone is standard, the reference speaking speed is 100%, and the reference volume is 80%.

[0089] There is no matching user identifier in the sample user identifier that matches the target user identifier, which means that the user identifier, age, gender and other data of the target user are not involved in the training process of the tone classification model, speech rate regression model and volume regression model. The tone classification model, speech rate regression model and volume regression model are difficult to accurately predict the voice parameters based on the user identifier, age and gender and other data of the target user. At this time, there is no need to combine the target scene data to predict the target tone, target volume and target speech rate. The reference tone, reference speech rate and reference volume can be directly obtained to generate the target speech according to the reference tone, reference speech rate and reference volume, thereby avoiding the situation where the tone classification model, speech rate regression model and volume regression model are difficult to accurately predict based on the target scene data when there is no matching user identifier in the sample user identifier that matches the target user identifier, resulting in low accuracy of the predicted target tone, target volume and target speech rate, making the generated target speech difficult to meet the requirements, thereby making the generated target speech more in line with the requirements.

[0090] At the same time, the reference tone, reference speaking speed and reference volume are directly obtained to generate the target speech, which saves the time of the tone classification model, speaking speed regression model and volume regression model to process the target scene data respectively, greatly reduces the data processing amount and improves the generation efficiency of the target speech.

[0091] See attached Figure 4 , Figure 4 The following is a block diagram of a speech generation device according to an embodiment of the present application. The device 800 is used in an electronic device and includes:

[0092] An acquisition module 810 is configured to acquire target text and target scene data for generating a target speech; the target scene data is configured to indicate a playback scene for the target speech;

[0093] A data processing module 820 is configured to process the target scene data using a tone classification model, a speech rate regression model, and a volume regression model, respectively, to obtain a target tone predicted by the tone classification model, a target speech rate predicted by the speech rate regression model, and a target volume predicted by the volume regression model. The tone classification model, the speech rate regression model, and the volume regression model are trained based on target losses, which are determined based on a first loss, a second loss, and a third loss. The first loss indicates the accuracy of the tone classification model's classification prediction, the second loss indicates the accuracy of the speech rate regression model's regression prediction, and the third loss indicates the accuracy of the volume regression model's regression prediction.

[0094] The speech generation module 830 is used to generate speech for the target text based on the target tone, target speech speed and target volume to obtain the target speech.

[0095] Optionally, the data processing module 820 is also used to perform feature embedding processing on the target scene data through a large model to obtain target scene features; perform classification prediction of the tone based on the target scene features through a tone classification model to obtain a target tone; perform regression prediction of the speech rate based on the target scene features through a speech rate regression model to obtain a target speech rate; and perform regression prediction of the volume based on the target scene features through a volume regression model to obtain a target volume.

[0096] Optionally, the device also includes a training module for obtaining sample scene data and sample annotation data corresponding to the sample speech; the sample annotation data includes sample tone, sample speaking rate and sample volume of the sample speech; the tone classification model, the speaking rate regression model and the volume regression model are used to process the sample scene data respectively to obtain the predicted sample tone predicted by the tone classification model, the predicted sample speaking rate predicted by the speaking rate regression model and the predicted sample volume predicted by the volume regression model; based on the predicted sample tone and the sample tone, a first loss is determined; based on the predicted sample speaking rate and the sample speaking rate, a second loss is determined; based on the predicted sample volume and the sample volume, a third loss is determined; based on the first loss, the second loss and the third loss, a target loss is determined; the tone classification model, the speaking rate regression model and the volume regression model are trained by the target loss until the training end condition is met.

[0097] Optionally, the sample scene data includes a sample user identifier of a sample user in a sample playback environment of the sample voice; the target scene data includes a target user identifier of a target user in a target playback environment of the target voice; the data processing module 820 is also used to process the target scene data based on the tone classification model, the speech rate regression model and the volume regression model if there is a matching user identifier in the sample user identifier to obtain the target tone predicted by the tone classification model, the target speech rate predicted by the speech rate regression model and the target volume predicted by the volume regression model.

[0098] Optionally, the speech generation module 830 is also used to obtain a preset reference tone, reference speaking speed and reference volume if there is no matching user identifier in the sample user identifier that matches the target user identifier; and generate speech for the target text based on the reference tone, reference speaking speed and reference volume to obtain the target speech.

[0099] Optionally, the training module is further used to perform a weighted summation on the first loss, the second loss, and the third loss to obtain a target loss.

[0100] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0101] In addition, the functions in the various embodiments of the present application may be integrated into a single processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules.

[0102] In addition, the functions in the various embodiments of the present application may be integrated into a single processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules.

[0103] Please refer to Figure 5 , which shows a structural block diagram of an electronic device provided according to an embodiment of the present application. The electronic device 500 can be an electronic device capable of running applications, such as a smartphone, a tablet computer, an e-book, and a vehicle. The electronic device 500 in the present application may include one or more of the following components: a processor 510, a memory 520, and one or more applications. The one or more applications may be stored in the memory 520 and configured to be executed by one or more processors 510, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0104] The processor 510 may include one or more processing cores. The processor 510 utilizes various interfaces and circuits to connect various components within the electronic device 500. It executes instructions, programs, code sets, or instruction sets stored in the memory 520, and accesses data stored in the memory 520 to perform various functions and process data within the electronic device 500. Optionally, the processor 510 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 510 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 510 and may be implemented separately via a communications chip.

[0105] The memory 520 may include a random access memory (RAM) or a read-only memory (ROM). The memory 520 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 520 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat history data) created by the electronic device 500 during use.

[0106] In addition, the functions in the various embodiments of the present application may be integrated into a single processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules.

[0107] On the other hand, the present application also provides a computer-readable storage medium, which stores program code. The program code can be called by a processor to execute the method described in the above method embodiment.

[0108] The computer-readable storage medium can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a cluster of ROMs. Alternatively, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code for executing any of the method steps described above. The program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in a suitable form.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A speech generation method, characterized in that: The method comprises: Acquire target text and target scene data for generating a target speech; the target scene data is used to indicate a playback scene for the target speech; The target scene data is processed respectively by a tone classification model, a speech rate regression model, and a volume regression model to obtain a target tone classified and predicted by the tone classification model, a target speech rate regressed and predicted by the speech rate regression model, and a target volume regressed and predicted by the volume regression model; wherein the tone classification model, the speech rate regression model, and the volume regression model are trained based on a target loss, and the target loss is determined based on a first loss, a second loss, and a third loss; the first loss indicates the accuracy of the classification prediction of the tone classification model, the second loss indicates the accuracy of the regression prediction of the speech rate regression model, and the third loss indicates the accuracy of the regression prediction of the volume regression model; Based on the target tone, the target speaking speed, and the target volume, speech generation is performed on the target text to obtain the target speech.

2. The method according to claim 1, characterized in that The tone classification model, the speech rate regression model, and the volume regression model are respectively processed based on the target scene data to obtain the target tone predicted by the tone classification model, the target speech rate predicted by the speech rate regression model, and the target volume predicted by the volume regression model, including: Performing feature embedding processing on the target scene data through a large model to obtain target scene features; Performing classification prediction of the tone based on the target scene features using the tone classification model to obtain a target tone; Performing a regression prediction of the speaking rate based on the target scene features using the speaking rate regression model to obtain a target speaking rate; The volume regression model is used to perform regression prediction of the volume based on the target scene features to obtain the target volume.

3. The method according to claim 1, characterized in that Before the tone classification model, the speech rate regression model, and the volume regression model are used to process the target scene data respectively to obtain the target tone predicted by the tone classification model, the target speech rate predicted by the speech rate regression model, and the target volume predicted by the volume regression model, the method further includes: Obtaining sample scene data and sample annotation data corresponding to the sample speech; the sample annotation data includes the sample tone, sample speaking speed and sample volume of the sample speech; The tone classification model, the speech rate regression model, and the volume regression model are used to process the sample scene data respectively, thereby obtaining a predicted sample tone predicted by the tone classification model, a predicted sample speech rate predicted by the speech rate regression model, and a predicted sample volume predicted by the volume regression model; determining a first loss based on the predicted sample tone and the sample tone; determining a second loss based on the predicted sample speech rate and the sample speech rate; determining a third loss based on the predicted sample volume and the sample volume; determining the target loss based on the first loss, the second loss, and the third loss; The tone classification model, the speech rate regression model, and the volume regression model are trained using the target loss until a training end condition is reached.

4. The method according to claim 3, characterized in that The sample scene data includes a sample user identifier of a sample user in a sample playback environment of the sample voice; the target scene data includes a target user identifier of a target user in a target playback environment of the target voice; The tone classification model, the speech rate regression model, and the volume regression model are respectively processed based on the target scene data to obtain the target tone predicted by the tone classification model, the target speech rate predicted by the speech rate regression model, and the target volume predicted by the volume regression model, including: If there is a matching user identifier in the sample user identifier that matches the target user identifier, the tone classification model, the speaking rate regression model and the volume regression model are respectively processed based on the target scene data to obtain the target tone predicted by the tone classification model, the target speaking rate predicted by the speaking rate regression model and the target volume predicted by the volume regression model.

5. The method according to claim 4, characterized in that The method further comprises: If there is no matching user identifier in the sample user identifiers that matches the target user identifier, obtaining a preset reference tone, reference speaking speed, and reference volume; Based on the reference tone, the reference speaking speed, and the reference volume, speech generation is performed on the target text to obtain the target speech.

6. The method according to claim 3, characterized in that The determining the target loss based on the first loss, the second loss, and the third loss includes: A weighted sum is performed on the first loss, the second loss, and the third loss to obtain the target loss.

7. The method according to any one of claims 1 to 6, characterized in that The target scene data includes the target user identifier of the target user in the target playback environment of the target voice, the age of the target user, the gender of the target user, the environment description information of the target playback environment, the dialogue scene information of the target playback environment and the context information of the target text.

8. A speech generating device, characterized in that: The device comprises: An acquisition module, configured to acquire target text and target scene data for generating a target speech; the target scene data being used to indicate a playback scene targeted by the target speech; a data processing module for processing the target scene data using a tone classification model, a speech rate regression model, and a volume regression model, respectively, to obtain a target tone predicted by the tone classification model, a target speech rate predicted by the speech rate regression model, and a target volume predicted by the volume regression model; wherein the tone classification model, the speech rate regression model, and the volume regression model are trained based on a target loss, and the target loss is determined based on a first loss, a second loss, and a third loss; the first loss indicates the accuracy of the classification prediction of the tone classification model, the second loss indicates the accuracy of the regression prediction of the speech rate regression model, and the third loss indicates the accuracy of the regression prediction of the volume regression model; The speech generation module is used to generate speech for the target text based on the target tone, the target speech speed and the target volume to obtain the target speech.

9. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program codes executable by a processor, and when the program codes are executed by the processor, the processor is caused to perform the method according to any one of claims 1 to 7.

Images