Voice reply method and device, electronic equipment and storage medium

Abstract

According to the voice reply method and device, the electronic equipment and the storage medium provided by the invention, the to-be-processed voice is subjected to voice conversion processing to obtain the to-be-processed text, and emotion classification processing is performed according to the to-be-processed voice and the to-be-processed text to obtain the first emotion; performing text classification processing on the to-be-processed text to obtain a text type, and determining a target reply text according to the text type and the first emotion; inputting the target reply text into a preset voice synthesis model for voice synthesis processing to obtain a target reply voice, and performing voice recognition processing on the target reply voice to obtain a reply voice text; and calculating the similarity between the reply voice text and the to-be-processed text to obtain a first similarity value, and taking the target reply voice as a final reply voice when the first similarity value is greater than a preset similarity threshold. Compared with the prior art, the embodiment of the invention can realize intelligent automatic reply, improve the accuracy and personification degree, and improve the user experience.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method and device for voice reply, an electronic device, and a storage medium. Background Art

[0002] The intelligent voice automatic reply function is an important application in modern communication technology. It uses artificial intelligence technology, especially voice recognition and natural language processing technology, to realize the function of automatically answering and replying to voice messages.

[0003] The intelligent voice automatic reply function originated from the traditional voice response system (Interactive Voice Response, IVR). The voice response system is mostly used in the customer service field. After the user dials the service number, the system will automatically answer the call and guide the user to select the required service through voice commands or buttons. With the advancement of artificial intelligence technology, modern intelligent voice automatic reply systems can understand more complex voice commands and natural languages, and make more natural responses.

[0004] However, the existing intelligent voice automatic reply system has added manual judgment and is a semi-intelligent automatic reply. It does not have sentiment analysis and has a single way of processing input voice. Therefore, it has limitations in processing complex contexts and expressing emotions. For example, it is not intelligent enough, the reply content is stiff, and lacks personalization, resulting in a poor user experience. Summary of the invention

[0005] The present disclosure provides a method, device, electronic device and storage medium for voice reply. Its main purpose is to solve the existing intelligent voice automatic reply system, which adds manual judgment, belongs to semi-intelligent automatic reply, has no emotion analysis, and has a single way of processing input voice. Therefore, there are limitations in processing complex contexts and expressing emotions, such as: not intelligent enough, stiff reply content, lack of personalization, etc., resulting in poor user experience.

[0006] According to a first aspect of the present disclosure, a method for voice reply is provided, comprising:

[0007] Performing voice conversion processing on the received speech to be processed to obtain a text to be processed, and performing emotion classification processing on the speech to be processed and the text to be processed to obtain a first emotion corresponding to the speech to be processed;

[0008] Performing text classification processing on the text to be processed to obtain a text type corresponding to the text to be processed, and determining a target reply text corresponding to the text to be processed according to the text type and the first emotion; wherein the text type includes sensitive text, operation text, marketing text, and other text;

[0009] Inputting the target reply text into a preset speech synthesis model for speech synthesis processing to obtain a target reply speech, and performing speech recognition processing on the target reply speech to obtain a reply speech text corresponding to the target reply speech;

[0010] The similarity between the reply voice text and the text to be processed is calculated to obtain a first similarity value, and when the first similarity value is greater than a preset similarity threshold, the target reply voice is used as the final reply voice corresponding to the voice to be processed.

[0011] Optionally, if the text type is the operation text, determining the target reply text corresponding to the to-be-processed text according to the text type and the first emotion includes:

[0012] Acquire first type information of a target user; wherein the target user is a target who sends the voice to be processed and to whom a voice reply is to be made;

[0013] Performing matching processing in a first preset database according to the first type of information and the text to be processed, obtaining a first reply candidate set; wherein the first reply candidate set includes at least one first reply text;

[0014] When the number of the first reply texts in the first reply candidate set is greater than a preset number threshold, respectively calculating the similarity between the to-be-processed text and each of the first reply texts in the first reply candidate set to obtain a second similarity value;

[0015] Compare each of the second similarity values ​​to obtain a maximum similarity value among all the second similarity values, and use the first reply text corresponding to the maximum similarity value as the first initial reply text;

[0016] According to the first emotion, matching processing is performed from different preset emotion databases to obtain a target emotion database, and according to the first initial reply text, matching processing is performed in the target emotion database to obtain a first modal particle; wherein different preset emotion databases correspond to different emotion categories;

[0017] The first initial reply text and the first modal particle are combined to obtain the target reply text.

[0018] Optionally, performing matching processing in a first preset database according to the first type of information and the text to be processed to obtain a first reply candidate set includes:

[0019] In the case where it is determined according to the first type of information that the target user is a new user, matching the to-be-processed text in the first operation database to obtain the first reply candidate set; wherein the target users are at least divided into the new users and the old users;

[0020] When it is determined that the target user is an old user based on the first type of information, the text to be processed is matched in the second operation database to obtain the first reply candidate set, wherein the first preset database includes the first operation database and the second operation database, and the data in the first operation database is different from the data in the second operation database.

[0021] Optionally, if the text type is the marketing text, determining the target reply text corresponding to the to-be-processed text according to the text type and the first emotion includes:

[0022] Acquire the user type of the target user and the region information corresponding to the target user; wherein the target user is the target who sends the voice to be processed and to whom a voice reply is to be made;

[0023] Perform matching processing in a second preset database according to the user type, the regional information and the text to be processed, and obtain a second reply text corresponding to the text to be processed;

[0024] According to the second reply text, matching processing is performed in a first preset modal particle database to obtain a second modal particle; wherein the first modal particle database is a pre-configured database containing a plurality of the second modal particles;

[0025] The second reply text and the second modal particle are combined to obtain the target reply text.

[0026] Optionally, if the text type is the other text, determining the target reply text corresponding to the to-be-processed text according to the text type and the first emotion includes:

[0027] Inputting the text to be processed into a preset text recognition model for text prediction processing to obtain a second reply candidate set corresponding to the text to be processed; wherein the second reply candidate set contains at least one third reply text;

[0028] Performing sentiment classification processing on each of the third reply texts in the second reply candidate set respectively to obtain a second sentiment corresponding to each of the third reply texts;

[0029] Calculating the emotion similarity between the first emotion and each of the second emotions respectively to obtain emotion similarity values, and performing comparison processing on each of the emotion similarity values ​​to obtain the maximum emotion similarity value among all the emotion similarity values;

[0030] Using the third reply text corresponding to the maximum sentiment similarity value as the second initial reply text;

[0031] Acquire the region information corresponding to the target user, and perform matching processing in the second preset modal particle database according to the region information to obtain the third modal particle; wherein the target user is the target who sends the voice to be processed and to be responded to by voice, and the second modal particle database is a pre-configured database containing a plurality of the third modal particles;

[0032] The second initial reply text and the third modal particle are combined to obtain the target reply text.

[0033] Optionally, if the text type is the sensitive text, determining the target reply text corresponding to the to-be-processed text according to the text type and the first emotion includes:

[0034] A fourth reply text is extracted from a third preset database as the target reply text, wherein the fourth reply text is a preset reply text, and the third preset database contains at least one fourth reply text.

[0035] Optionally, the step of inputting the target reply text into a preset speech synthesis model for speech synthesis processing to obtain the target reply speech includes:

[0036] Based on the preset speech synthesis model, performing text analysis processing on the target reply text to obtain context attributes of the target reply text;

[0037] The target reply text is clustered and decided according to the context attributes and the clustering decision tree to obtain model sequence data; wherein the preset speech synthesis model includes the clustering decision tree, and the clustering decision tree includes at least a fundamental frequency decision tree, a spectrum decision tree and a duration decision tree;

[0038] Performing parameter generation processing according to the model sequence data to obtain fundamental frequency parameters, spectrum parameters and duration parameters corresponding to the target reply text; wherein the model sequence data at least includes fundamental frequency data, spectrum data and duration data of the target reply text;

[0039] Speech synthesis processing is performed according to the fundamental frequency parameter, the spectrum parameter and the duration parameter to obtain the target reply speech.

[0040] Optionally, performing emotion classification processing according to the speech to be processed and the text to be processed to obtain a first emotion corresponding to the speech to be processed includes:

[0041] Preprocessing the text to be processed to obtain a word vector corresponding to the text to be processed, and preprocessing the speech to be processed to obtain a speech frame corresponding to the speech to be processed;

[0042] Inputting the word vector into a preset semantic recognition model for semantic recognition processing to obtain semantic features of the text to be processed, and inputting the speech frame into a preset audio recognition model for audio recognition processing to obtain audio features of the speech to be processed;

[0043] Inputting the semantic features into a preset emotion classification model for emotion classification processing to obtain the semantic emotion of the text to be processed, and inputting the audio features into the preset emotion classification model for emotion classification processing to obtain the audio emotion of the speech to be processed;

[0044] The semantic emotion is fused with the audio emotion to obtain the first emotion.

[0045] Optionally, before performing voice conversion processing on the received voice to be processed to obtain the text to be processed, the method further includes:

[0046] Modifying the dependency of the observation value parameters in the initial speech synthesis model according to the adjacent time state of the initial speech synthesis model to obtain a modified initial speech synthesis model; wherein the adjacent time state includes the model state of the initial speech synthesis model at the previous adjacent time of the current time, and the model state of the initial speech synthesis model at the next adjacent time of the current time;

[0047] The initial state probability matrix in the modified initial speech synthesis model is modified according to the adjacent time states to obtain the speech synthesis model to be trained.

[0048] Optionally, after modifying the initial state probability matrix in the modified initial speech synthesis model according to the adjacent time states to obtain the speech synthesis model to be trained, the method further includes:

[0049] Acquire a first preset amount of first training data and training context attributes; wherein the first training data at least includes training acoustic parameters, and the training context attributes are context attributes that have an impact on the training acoustic parameters;

[0050] Performing model training processing on the speech synthesis model to be trained using the first training data to obtain a single factor state of the speech synthesis model to be trained, and performing model training processing on the speech synthesis model to be trained using the training context attributes to obtain a context state of the speech synthesis model to be trained;

[0051] Performing decision clustering processing on the speech synthesis model to be trained by a preset decision tree clustering algorithm to obtain the clustering state of the speech synthesis model to be trained, and obtaining the first residence time of the single factor state, the second residence time of the context state, and the third residence time of the cluster state;

[0052] Initializing a preset duration model according to the first residence time, the second residence time, and the third residence time to obtain an initial duration model;

[0053] The preset decision tree clustering algorithm is used to perform model clustering processing on the speech synthesis model to be trained and the initial duration model to obtain the preset speech synthesis model.

[0054] Optionally, after calculating the similarity between the reply voice text and the text to be processed to obtain a first similarity value, the method further includes:

[0055] When the first similarity value is less than or equal to the preset similarity threshold, obtaining a second preset amount of second training data;

[0056] The preset speech synthesis model is subjected to model training processing according to the reply speech text and the second training data to obtain an updated speech synthesis model.

[0057] According to a second aspect of the present disclosure, a device for voice reply is provided, comprising:

[0058] A conversion unit, used for performing speech conversion processing on the received speech to be processed to obtain a text to be processed;

[0059] A first classification unit is used to perform emotion classification processing according to the speech to be processed and the text to be processed to obtain a first emotion corresponding to the speech to be processed;

[0060] A second classification unit is used to perform text classification processing on the text to be processed to obtain a text type corresponding to the text to be processed;

[0061] A first determination unit is used to determine a target reply text corresponding to the to-be-processed text according to the text type and the first emotion; wherein the text type includes sensitive text, operation text, marketing text, and other text;

[0062] A synthesis unit, used for inputting the target reply text into a preset speech synthesis model for speech synthesis processing to obtain a target reply speech;

[0063] A recognition unit, configured to perform speech recognition processing on the target reply speech to obtain a reply speech text corresponding to the target reply speech;

[0064] A calculation unit, used for calculating the similarity between the reply voice text and the text to be processed to obtain a first similarity value;

[0065] The second determining unit is used to use the target reply voice as the final reply voice corresponding to the to-be-processed voice when the first similarity value is greater than a preset similarity threshold.

[0066] Optionally, the first determining unit includes:

[0067] An acquisition module, used to acquire first type information of a target user; wherein the target user is a target who sends the voice to be processed and to whom a voice reply is to be made;

[0068] A matching module, configured to perform matching processing in a first preset database according to the first type of information and the text to be processed, to obtain a first reply candidate set; wherein the first reply candidate set contains at least one first reply text;

[0069] A calculation module, configured to respectively calculate the similarity between the to-be-processed text and each of the first reply texts in the first reply candidate set to obtain a second similarity value when the number of the first reply texts in the first reply candidate set is greater than a preset number threshold;

[0070] a comparison module, configured to compare each of the second similarity values ​​to obtain a maximum similarity value among all the second similarity values, and use the first reply text corresponding to the maximum similarity value as a first initial reply text;

[0071] The matching module is further used to perform matching processing from different preset emotion databases according to the first emotion to obtain a target emotion database, and perform matching processing in the target emotion database according to the first initial reply text to obtain a first modal particle; wherein different preset emotion databases correspond to different emotion classifications;

[0072] A merging module is used to merge the first initial reply text with the first modal particle to obtain the target reply text.

[0073] Optionally, the matching module is further used for:

[0074] In the case where it is determined according to the first type of information that the target user is a new user, matching the to-be-processed text in the first operation database to obtain the first reply candidate set; wherein the target users are at least divided into the new users and the old users;

[0075] When it is determined that the target user is an old user based on the first type of information, the text to be processed is matched in the second operation database to obtain the first reply candidate set, wherein the first preset database includes the first operation database and the second operation database, and the data in the first operation database is different from the data in the second operation database.

[0076] Optionally, the first determining unit is further configured to:

[0077] Acquire the user type of the target user and the region information corresponding to the target user; wherein the target user is the target who sends the voice to be processed and to whom a voice reply is to be made;

[0078] Perform matching processing in a second preset database according to the user type, the regional information and the text to be processed, and obtain a second reply text corresponding to the text to be processed;

[0079] According to the second reply text, matching processing is performed in a first preset modal particle database to obtain a second modal particle; wherein the first modal particle database is a pre-configured database containing a plurality of the second modal particles;

[0080] The second reply text and the second modal particle are combined to obtain the target reply text.

[0081] Optionally, the first determining unit is further configured to:

[0082] Inputting the text to be processed into a preset text recognition model for text prediction processing to obtain a second reply candidate set corresponding to the text to be processed; wherein the second reply candidate set contains at least one third reply text;

[0083] Performing sentiment classification processing on each of the third reply texts in the second reply candidate set respectively to obtain a second sentiment corresponding to each of the third reply texts;

[0084] Calculating the emotion similarity between the first emotion and each of the second emotions respectively to obtain emotion similarity values, and performing comparison processing on each of the emotion similarity values ​​to obtain the maximum emotion similarity value among all the emotion similarity values;

[0085] Using the third reply text corresponding to the maximum sentiment similarity value as the second initial reply text;

[0086] Acquire the region information corresponding to the target user, and perform matching processing in the second preset modal particle database according to the region information to obtain the third modal particle; wherein the target user is the target who sends the voice to be processed and to be responded to by voice, and the second modal particle database is a pre-configured database containing a plurality of the third modal particles;

[0087] The second initial reply text and the third modal particle are combined to obtain the target reply text.

[0088] Optionally, the first determining unit is further configured to:

[0089] A fourth reply text is extracted from a third preset database as the target reply text, wherein the fourth reply text is a preset reply text, and the third preset database contains at least one fourth reply text.

[0090] Optionally, the synthesis unit comprises:

[0091] An analysis module, configured to perform text analysis on the target reply text based on the preset speech synthesis model to obtain context attributes of the target reply text;

[0092] A decision module, used for performing clustering decision processing on the target reply text according to the context attributes and the clustering decision tree to obtain model sequence data; wherein the preset speech synthesis model includes the clustering decision tree, and the clustering decision tree at least includes a fundamental frequency decision tree, a spectrum decision tree and a duration decision tree;

[0093] A generation module, used to perform parameter generation processing according to the model sequence data to obtain fundamental frequency parameters, spectrum parameters and duration parameters corresponding to the target reply text; wherein the model sequence data at least includes fundamental frequency data, spectrum data and duration data of the target reply text;

[0094] A synthesis module is used to perform speech synthesis processing according to the fundamental frequency parameters, the spectrum parameters and the duration parameters to obtain the target reply speech.

[0095] Optionally, the first classification unit includes:

[0096] A processing module, used to perform a preprocessing operation on the text to be processed to obtain a word vector corresponding to the text to be processed, and to perform a preprocessing operation on the speech to be processed to obtain a speech frame corresponding to the speech to be processed;

[0097] A recognition module, used for inputting the word vector into a preset semantic recognition model for semantic recognition processing to obtain semantic features of the text to be processed, and inputting the speech frame into a preset audio recognition model for audio recognition processing to obtain audio features of the speech to be processed;

[0098] A classification module, used for inputting the semantic features into a preset emotion classification model for emotion classification processing to obtain the semantic emotion of the text to be processed, and inputting the audio features into the preset emotion classification model for emotion classification processing to obtain the audio emotion of the speech to be processed;

[0099] A fusion module is used to fuse the semantic emotion with the audio emotion to obtain the first emotion.

[0100] Optionally, the device further comprises:

[0101] A modification unit, used for modifying the dependency of the observation value parameters in the initial speech synthesis model according to the adjacent time state of the initial speech synthesis model, so as to obtain a modified initial speech synthesis model; wherein the adjacent time state includes the model state of the initial speech synthesis model at the previous adjacent time of the current time, and the model state of the initial speech synthesis model at the next adjacent time of the current time;

[0102] The modification unit is further used to modify the initial state probability matrix in the modified initial speech synthesis model according to the adjacent time states to obtain the speech synthesis model to be trained.

[0103] Optionally, the device further comprises:

[0104] An acquisition unit, configured to acquire a first preset amount of first training data and training context attributes; wherein the first training data at least includes training acoustic parameters, and the training context attributes are context attributes that have an impact on the training acoustic parameters;

[0105] A training unit, configured to perform model training processing on the speech synthesis model to be trained using the first training data to obtain a single factor state of the speech synthesis model to be trained, and to perform model training processing on the speech synthesis model to be trained using the training context attributes to obtain a context state of the speech synthesis model to be trained;

[0106] The training unit is further used to perform decision clustering processing on the speech synthesis model to be trained by using a preset decision tree clustering algorithm to obtain a clustering state of the speech synthesis model to be trained;

[0107] The acquisition unit is further used to acquire a first stay time of the single factor state, a second stay time of the context state, and a third stay time of the cluster state;

[0108] a processing unit, configured to initialize a preset duration model according to the first residence time, the second residence time, and the third residence time to obtain an initial duration model;

[0109] The training unit is also used to perform model clustering processing on the speech synthesis model to be trained and the initial duration model through the preset decision tree clustering algorithm to obtain the preset speech synthesis model.

[0110] Optionally, the acquisition unit is further configured to, when the first similarity value is less than or equal to the preset similarity threshold, acquire a second preset amount of second training data;

[0111] The training unit is also used to perform model training processing on the preset speech synthesis model according to the reply voice text and the second training data to obtain an updated speech synthesis model.

[0112] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0113] at least one processor; and

[0114] a memory communicatively connected to the at least one processor; wherein,

[0115] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0116] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0117] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described in the first aspect above.

[0118] The method and device, electronic device and storage medium for voice reply provided by the present disclosure perform voice conversion processing on the received voice to be processed to obtain text to be processed, and perform emotion classification processing on the voice to be processed and the text to be processed to obtain a first emotion corresponding to the voice to be processed; perform text classification processing on the text to be processed to obtain a text type corresponding to the text to be processed, and determine a target reply text corresponding to the text to be processed according to the text type and the first emotion; wherein the text type includes sensitive text, operation text, marketing text, and other text; input the target reply text into a preset voice synthesis model for voice synthesis processing to obtain a target reply voice, and perform voice recognition processing on the target reply voice to obtain a reply voice text corresponding to the target reply voice; calculate the similarity between the reply voice text and the text to be processed to obtain a first similarity value, and when the first similarity value is greater than a preset similarity threshold, use the target reply voice as the final reply voice corresponding to the voice to be processed. Compared with the related art, the embodiment of the present disclosure converts the received voice to be processed into text data (text to be processed), classifies the text data, determines the text emotion (first emotion) and text type of the text data, obtains reply data (target reply text) corresponding to different types of text data and text emotions, replies to the target reply text with voice (target reply voice), and then converts it into text (reply voice text), compares the reply voice text with the initial target reply text, and when the two texts have a high degree of similarity, uses the target reply voice as the final voice for reply, thereby realizing intelligent automatic reply without human intervention, improving the accuracy and degree of anthropomorphism, and enhancing the user experience.

[0119] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0120] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0121] Figure 1 A flowchart of a voice reply method provided by an embodiment of the present disclosure;

[0122] Figure 2 A flowchart of a method for determining a target reply text provided by an embodiment of the present disclosure;

[0123] Figure 3 A schematic diagram of a construction process of a first preset database provided in an embodiment of the present disclosure;

[0124] Figure 4 A flowchart of another method for determining a target reply text provided by an embodiment of the present disclosure;

[0125] Figure 5 A flowchart of another method for determining a target reply text provided by an embodiment of the present disclosure;

[0126] Figure 6 A schematic diagram of a sentiment classification process provided by an embodiment of the present disclosure;

[0127] Figure 7 A schematic diagram of another emotion classification process provided by an embodiment of the present disclosure;

[0128] Figure 8 A flowchart of a method for acquiring a speech synthesis model to be trained provided by an embodiment of the present disclosure;

[0129] Fig. 9 A schematic diagram of a training process of a preset speech synthesis model provided in an embodiment of the present disclosure;

[0130] Fig.10 A flowchart of another voice reply method provided by an embodiment of the present disclosure;

[0131] Fig.11 A schematic diagram of the structure of a voice reply device provided by an embodiment of the present disclosure;

[0132] Fig.12 A schematic diagram of the structure of another voice reply device provided by an embodiment of the present disclosure;

[0133] Fig.13 A schematic block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0134] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0135] The following describes the voice reply method and device, electronic device, and storage medium of the embodiments of the present disclosure with reference to the accompanying drawings.

[0136] Figure 1 A flowchart of a voice reply method provided in an embodiment of the present disclosure.

[0137] like Figure 1 As shown, the method comprises the following steps:

[0138] Step 101, performing speech conversion processing on the received speech to be processed to obtain text to be processed, and performing emotion classification processing on the speech to be processed and the text to be processed to obtain a first emotion corresponding to the speech to be processed.

[0139] In the embodiment of the present disclosure, the conversion method for the speech to be processed is a custom-set algorithm, such as the textrank algorithm, etc., which is used to perform keyword extraction processing on the speech to be processed to obtain the text to be processed. Specifically, the embodiment of the present disclosure does not limit the selection of the conversion method for the speech to be processed.

[0140] Therefore, the textrank algorithm can be used to perform unsupervised keyword extraction on the speech to be processed to obtain the text to be processed. Specifically, the processing of the speech to be processed by the textrank algorithm can be implemented by, but not limited to, formula (1):

[0141]

[0142] Among them, WS(V i ) represents the weight of the sentence (speech to be processed), and the sum on the right represents the contribution of each adjacent sentence (speech adjacent to the speech to be processed) to the sentence (speech to be processed). Since in a single document, it can be roughly considered that all sentences are adjacent, there is no need to generate and extract multiple windows like multiple documents. Therefore, only a single document window is needed, that is, only the weight of the speech to be processed needs to be calculated, W ji represents the similarity between two sentences (the speech to be processed and the speech adjacent to the speech to be processed), WS(V j ) represents the weight of the sentence iterated last time (the speech adjacent to the speech to be processed), d is the damping coefficient, and the damping coefficient is a coefficient set by a user, for example: 0.75, 0.85, etc.

[0143] In summary, regarding the speech conversion processing of the speech to be processed, it can be implemented by but not limited to the following methods: performing weight calculation processing on the speech to be converted through a preset keyword extraction algorithm to obtain the keyword weight in the speech to be converted; performing keyword extraction processing on the speech to be converted according to the keyword weight to obtain keyword data; performing text generation processing according to the keyword data to obtain the text to be processed.

[0144] The method for performing emotion classification processing is a custom setting method, for example: a bimodal speech emotion classifier, etc. Specifically, the method for performing emotion classification processing can be customized and selected, and the embodiments of the present disclosure do not limit it.

[0145] Specifically, regarding sentiment classification processing, it can be implemented by but not limited to the following methods: using a classifier that integrates semantic and audio bimodal speech sentiment classification, after converting speech (speech to be processed) to text (text to be processed), preprocessing the speech and text, inputting the preprocessed text into a bidirectional long short-term memory network (BidirectionalLong Short-Term Memory, BLSTM) to obtain semantic features, sending the semantic features into a trained convolutional neural network (Convolutional Neural Network, CNN) semantic sentiment classifier to obtain semantic sentiment classification, using the OpenSmille tool to obtain the acoustic features of the speech, sending the acoustic features into a trained CNN audio sentiment classifier to obtain the sentiment classification of the speech, performing decision-making layer fusion on the semantic sentiment classification results and the speech sentiment classification results, calculating the score of each emotion through the decision-making layer fusion formula, and taking the highest score as the final bimodal speech sentiment classification result (first emotion).

[0146] Step 102, performing text classification processing on the text to be processed to obtain the text type corresponding to the text to be processed, and determining the target reply text corresponding to the text to be processed according to the text type and the first emotion; wherein the text type includes sensitive text, operation text, marketing text, and other text.

[0147] In the embodiment of the present disclosure, when performing text classification processing on the text to be processed, the text to be processed can be divided into four categories according to the keywords of the text to be processed: text containing sensitive words (sensitive text), text containing operation manual keywords (operation text), text containing keywords such as event discount marketing (marketing text), and other texts.

[0148] Among them, after the classification processing, different methods will be selected to determine the target reply text according to the different types of the text to be processed. Regarding the method of determining the target reply text, the following methods can be adopted but are not limited to: when the text (text to be processed) is classified, different types of text are processed differently. For texts containing sensitive words (sensitive texts), different categories of safe replies are set according to the speech emotion classification results (first emotion) to determine the target reply text; for texts containing professional terms (for example: containing software operation words) (operation texts), the operation manual is used to preset the reply library, and combined with the speech emotion classification results, modal particles are added to the reply to increase the temperature of the reply, and the target reply text is obtained; for texts containing keywords such as event discount marketing (marketing texts), according to the classification of user types, province classification, and user experience time, combined with the speech emotion classification results, a promotion reply for the corresponding activity is given to obtain the target reply text; for other texts, the speech emotion classification results are compared with the results obtained after semantic emotion classification of the reply candidate set obtained through the long short-term memory (LSTM) network model, and the candidate reply with the highest emotional similarity is obtained as the target reply text.

[0149] Step 103: input the target reply text into a preset speech synthesis model for speech synthesis processing to obtain a target reply speech, and perform speech recognition processing on the target reply speech to obtain a reply speech text corresponding to the target reply speech.

[0150] In the embodiment of the present disclosure, the preset speech synthesis model is a custom selected model, for example: a dynamic tts model based on an improved HMM, etc. Specifically, the embodiment of the present disclosure does not limit the preset speech synthesis model.

[0151] Therefore, regarding the speech synthesis processing of the target reply text, the following method can be adopted but is not limited to: using a dynamic text-to-speech (tts) model based on an improved hidden Markov model (HMM), firstly predicting the parameters of the dynamic tts model, using HMM training to model the spectrum, fundamental frequency and duration parameters, and producing respective state decision trees; using decision trees to predict the model, and then generating parameters according to dynamic characteristics to obtain a trained dynamic tts model based on the improved HMM. Then, based on the HMM, the duration, fundamental frequency and spectrum parameters of the target reply text are predicted, and at the same time, the attribute analysis of the target reply text is performed, and finally a parameter synthesizer is used to synthesize the target reply speech.

[0152] Among them, when performing speech recognition processing on the target reply voice, it can be adopted but not limited to: reverse outputting the target reply voice through an automatic speech recognition (Automatic Speech Recognition, ASR) model to obtain the corresponding text of the target reply voice (reply voice text).

[0153] Step 104, calculate the similarity between the reply voice text and the text to be processed to obtain a first similarity value, and when the first similarity value is greater than a preset similarity threshold, use the target reply voice as the final reply voice corresponding to the voice to be processed.

[0154] In the embodiment of the present disclosure, the method for calculating the first similarity value is a user-selected method, such as calculation of cosine similarity, etc. Specifically, the embodiment of the present disclosure does not limit the method for calculating the first similarity value.

[0155] It should be noted that when calculating the first similarity value, it is necessary to convert the reply voice text and the text to be processed into vector form in advance and then perform the calculation. Therefore, the calculation of the similarity between the reply voice text and the text to be processed can be carried out in the following manner but is not limited to: performing data conversion processing on the text to be processed to obtain the first vector corresponding to the text to be processed, and performing data conversion processing on the reply voice text to obtain the second vector corresponding to the reply voice text; calculating the similarity between the first vector and the second vector through a preset similarity algorithm to obtain the first similarity value.

[0156] Meanwhile, the calculation of the first similarity value may be implemented by, but not limited to, formula (2):

[0157]

[0158] Wherein, cos(x,x) is the cosine similarity between the reply voice text and the text to be processed, that is, the first similarity value, x1 is the first vector corresponding to the text to be processed, and x2 is the second vector corresponding to the reply voice text. It should be noted that if the cosine value cos(x,x) (first similarity value) is closer to 1, it means that the angle between the first vector and the second vector is closer to 0, and the two vectors are more similar, that is, the reply voice text is more similar to the text to be processed; if the cosine value cos(x,x) (first similarity value) is closer to 0, it means that the angle between the first vector and the second vector is closer to 90 degrees, and the two vectors are less similar, that is, the reply voice text is less similar to the text to be processed.

[0159] When determining whether the first similarity value is greater than the preset similarity threshold, the preset similarity threshold is a custom-set value, such as 0.8, 0.9, etc. Specifically, the embodiment of the present disclosure does not limit the setting of the preset similarity threshold.

[0160] The method for voice reply provided by the present disclosure performs voice conversion processing on the received voice to be processed to obtain a text to be processed, and performs emotion classification processing on the voice to be processed and the text to be processed to obtain a first emotion corresponding to the voice to be processed; performs text classification processing on the text to be processed to obtain a text type corresponding to the text to be processed, and determines a target reply text corresponding to the text to be processed according to the text type and the first emotion; wherein the text type includes sensitive text, operation text, marketing text, and other text; inputs the target reply text into a preset voice synthesis model for voice synthesis processing to obtain a target reply voice, and performs voice recognition processing on the target reply voice to obtain a reply voice text corresponding to the target reply voice; calculates the similarity between the reply voice text and the text to be processed to obtain a first similarity value, and when the first similarity value is greater than a preset similarity threshold, uses the target reply voice as the final reply voice corresponding to the voice to be processed. Compared with the related art, the embodiment of the present disclosure converts the received voice to be processed into text data (text to be processed), classifies the text data, determines the text emotion (first emotion) and text type of the text data, obtains reply data (target reply text) corresponding to different types of text data and text emotions, replies to the target reply text with voice (target reply voice), and then converts it into text (reply voice text), compares the reply voice text with the initial target reply text, and when the two texts have a high degree of similarity, uses the target reply voice as the final voice for reply, thereby realizing intelligent automatic reply without human intervention, improving the accuracy and degree of anthropomorphism, and enhancing the user experience.

[0161] In one possible implementation of the embodiment of the present disclosure, due to the different types of the text to be processed, the target reply text will be determined in different ways. Therefore, it is necessary to explain the method of determining the target reply text in different situations. Regarding the determination of the target reply text, the embodiment of the present disclosure provides a flow chart of the method of determining the target reply text, as shown in FIG. Figure 2 As shown, including:

[0162] Step 201, obtaining first type information of a target user; wherein the target user is a target who sends the voice to be processed and to whom a voice reply is to be made.

[0163] In an embodiment of the present disclosure, the first type of information is used to determine the type of the target user, and the type of the target user includes old users and new users, wherein the old users and the new users can be distinguished by the usage time of the account corresponding to the target user. Specifically, it can be achieved by but not limited to the following methods: obtaining the usage time information of the account corresponding to the target user, wherein the account is bound to the target user and is used to log in to the voice reply system; determining the first type of information of the target user according to the usage time information.

[0164] If it is determined according to the usage time information that the usage time of the account exceeds the preset time threshold, the target user is determined to be an old user; if it is determined according to the usage time information that the usage time of the account does not exceed the preset time threshold, the target user is determined to be a new user. For example: the user experience value is determined according to the account usage time, and the user with a usage period of less than 1 year is determined to be a new user, and the user with a usage period of more than 1 year is determined to be an old user.

[0165] Step 202, matching processing is performed in a first preset database according to the first type of information and the text to be processed to obtain a first reply candidate set; wherein the first reply candidate set contains at least one first reply text.

[0166] In an embodiment of the present disclosure, the first reply text is a pre-set reply text, and the first preset database is a pre-set database containing multiple first reply texts. Depending on the type of the target user, matching is performed from different databases to obtain the first reply candidate set. For example: if the target user is determined to be a new user based on the first type of information, matching is performed from the new user reply text database to obtain the first reply candidate set; if the target user is determined to be an old user based on the first type of information, matching is performed from the old user reply text database to obtain the first reply candidate set.

[0167] The content of the new user reply text database is different from the content of the old user reply text database. The content of the new user reply text database is, for example, a library element object of {keyword set, question, manual answer}, and the content of the old user reply text database is, for example, a library element object of {keyword set, question, concise answer}. It can be understood that the content in the new user reply text database is more detailed than the content in the old user reply text database.

[0168] Step 203, when the number of the first reply texts in the first reply candidate set is greater than a preset number threshold, the similarity between the to-be-processed text and each of the first reply texts in the first reply candidate set is calculated respectively to obtain a second similarity value.

[0169] In the disclosed embodiment, since a plurality of first reply texts may be obtained by performing matching processing in a first preset database of keywords, it is necessary to calculate the similarity between the text to be processed and each first reply text so as to select the first reply text closest to the text to be processed as the target reply text.

[0170] The preset quantity threshold is a custom-set value, such as 1, 2, etc. The preset quantity threshold is generally set to 1, indicating that if the text to be processed matches more than one first reply text, the similarity between the text to be processed and each of the first reply texts is calculated.

[0171] The method for calculating the second similarity value is a user-selected method, such as calculation of cosine similarity, etc. Specifically, the embodiment of the present disclosure does not limit the method for calculating the second similarity value.

[0172] It should be noted that before calculating the second similarity value, it is also necessary to convert the text to be processed and the first reply text into vectors, for example: perform data conversion processing on each of the first reply texts respectively to obtain the third vector corresponding to each of the first reply texts; calculate the similarity between the first vector and the third vector through the preset similarity algorithm to obtain the second similarity value.

[0173] Regarding the calculation of the second similarity value, please refer to the description in the above step 104, so it will not be described in detail here.

[0174] Step 204: perform a comparison process on each of the second similarity values ​​to obtain a maximum similarity value among all the second similarity values, and use the first reply text corresponding to the maximum similarity value as the first initial reply text.

[0175] In the disclosed embodiment, it is necessary to select a corresponding library for matching based on the type of target user, and match a certain element object (first reply text) or several element objects in the database through keywords. If only one element object is matched, the answer to the question in the element object is taken out. If multiple elements are matched, the cosine similarity of the text input and the questions in several elements is calculated respectively, and the answer to the object element with the highest similarity is selected and taken out as the first initial reply text.

[0176] Step 205, matching processing is performed from different preset emotion databases according to the first emotion to obtain a target emotion database, and matching processing is performed in the target emotion database according to the first initial reply text to obtain a first modal particle; wherein different preset emotion databases correspond to different emotion classifications.

[0177] In the disclosed embodiment, the preset emotion database is a pre-configured database containing modal particles corresponding to emotion classifications. The preset emotion database is some custom-set databases. The types of the preset emotion databases include but are not limited to five categories: fear, anger, sadness, happiness, and disgust. The preset emotion database is a corresponding modal particle library created for each of the five emotion categories.

[0178] The target emotion database is a database matched from different types of preset emotion databases according to the first emotion, and then the first modal particle is matched from the target emotion database.

[0179] Step 206: merge the first initial reply text and the first modal particle to obtain the target reply text.

[0180] In the disclosed embodiment, according to the speech emotion classification result (first emotion), a suitable mood particle (first mood particle) is selected from the corresponding emotion category mood particle library (preset emotion database) to add the mood particle to the obtained answer (first initial reply text) as the target reply text. This can make the target reply text distinctive and enhance the user experience.

[0181] In one implementable manner of the embodiment of the present disclosure, as a refinement of the above-mentioned step 202, it can also be implemented in the following manner but not limited to: when it is determined that the target user is a new user based on the first type of information, the text to be processed is matched in the first operation database to obtain the first reply candidate set; wherein the target users are at least divided into the new users and the old users; when it is determined that the target user is an old user based on the first type of information, the text to be processed is matched in the second operation database to obtain the first reply candidate set, wherein the first preset database includes the first operation database and the second operation database, and the data in the first operation database is different from the data in the second operation database.

[0182] In the embodiment of the present disclosure, the first operation database is the new user reply text database, and the second operation database is the new user reply text database. Specifically, regarding the implementation process of the embodiment of the present disclosure, please refer to the description of step 202 above, so it will not be repeated here one by one.

[0183] In summary, when the text type is the operation text, the process of determining the target reply text can be achieved by but not limited to the following methods: if no match is found in the sensitive word library, the keywords of the text information (text to be processed) are matched with the standard operation manual response library (the first preset database). If a match is found, it is classified as a text containing operation manual keywords (determining that the text type is the operation text). The target users are divided into new and old users according to their years of use, for example: those with a usage period of less than 1 year are classified as new users, and those with a usage period of more than one year are classified as old users.

[0184] For new users, the keywords of the text to be processed are matched with the standard operation manual answer library (first operation database); for old users, the keywords of the text to be processed are matched with the popular operation manual answer library (second operation database); if a unique keyword element object is matched (there is only one first reply text in the first reply candidate set), the answer in the corresponding element object is returned; if multiple element objects are matched (there are multiple first reply texts in the first reply candidate set), the cosine similarity of the text to be processed and the matched element objects (first reply text) is calculated, and the element object with high similarity is taken as the first initial reply text; on the other hand, the trained CNN network model is used to perform emotion classification on the processed speech, which is divided into five categories: fear, anger, sadness, happiness, and disgust; according to the speech emotion classification result (first emotion) and the degree of emotion matching, appropriate mood particles (first mood particles) are selected from the corresponding emotion category mood particle library (target emotion database) to add mood particles to the first initial reply text; the higher the degree of emotion matching, the stronger the selected mood particles (first mood particles) can be, and the final result is used as the speech automatic reply text (target reply text).

[0185] It should be noted that before determining the target reply text, it is necessary to perform database construction processing on the first preset database. Regarding the construction of the first preset database, the following methods may also be used but are not limited to: extracting question data from a preset operation manual, and performing keyword extraction processing on the question data according to the preset keyword extraction algorithm to obtain question keyword data; extracting answer data from the preset operation manual, and merging the keyword data, the question data and the answer data to obtain the first reply text; performing database construction processing based on a preset number of the first reply texts to obtain the first preset database.

[0186] In order to facilitate understanding of the embodiment of the present disclosure, the embodiment of the present disclosure provides a schematic diagram of a construction process of a first preset database, such as Figure 3As shown, it includes: creating a standard operation manual answer library, the elements in the library are composed of an object {keyword set, question, manual answer}, traversing each question in the operation manual, and using textrank to extract keywords, and the obtained keywords, questions and answers constitute the library element object, and finally obtain the standard operation manual answer library (first operation database). The same method is used to create a popular operation manual answer library (second operation database), but the library element object is {keyword set, question, concise answer}, and the concise answer is obtained by polishing and simplifying the manual answer.

[0187] In one possible implementation of the embodiment of the present disclosure, the above describes the process of determining the target reply text when the text type is the operation text. Next, the process of determining the target reply text when the text type is the marketing text is described. When the text type is the marketing text, the present disclosure provides a flow chart of another method for determining the target reply text, such as Figure 4 As shown, including:

[0188] Step 401, obtaining the user type of a target user and the region information corresponding to the target user; wherein the target user is the target who sends the voice to be processed and to whom a voice reply is to be made.

[0189] In the disclosed embodiment, the user type at least includes the applicable activity type of the target user, etc., and the geographical information is the area where the target user is located, which is obtained according to the province of the target user and the geographical location area.

[0190] Step 402, performing matching processing in a second preset database according to the user type, the regional information and the text to be processed, to obtain a second reply text corresponding to the text to be processed.

[0191] In the disclosed embodiment, the second preset database includes a variety of library elements (second reply texts). The form of the second reply text includes but is not limited to: {marketing campaign name, applicable user type, province, start time, end time, activity rules}, etc., and is matched according to key information such as the user type of the target user and the province (regional information). All marketing activities that meet the text to be processed are combined into a complete activity introduction in the order of marketing campaign name, applicable user type, province, start time, end time, and activity rules to obtain the second reply text corresponding to the text to be processed.

[0192] Step 403, performing matching processing in a first preset modal particle database according to the second reply text to obtain a second modal particle; wherein the first modal particle database is a pre-configured database containing a plurality of the second modal particles.

[0193] In the embodiment of the present disclosure, the first preset modal particle database is a custom-set database, which contains a plurality of the second modal particles, and the second modal particles are custom-set modal particles, for example: happy modal particles (ha, hey, hi, etc.), lively modal particles (le, ah, oh, ne, etc.). Specifically, the embodiment of the present disclosure does not limit the second modal particles and the first preset modal particle database.

[0194] Step 404: merge the second reply text and the second modal particle to obtain the target reply text.

[0195] In the disclosed embodiment, appropriate mood particles (second mood particles) are selected from the corresponding happy mood particle library or lively mood particle library (first preset mood particle database) and added to the activity introduction (second reply text) as the target reply text. This can make the target reply text distinctive and enhance the user experience.

[0196] In summary, when the text type is the marketing text, the process of determining the target reply text can be achieved by but not limited to the following methods: if the sensitive word library and the standard operation manual answer library are not matched, but the text contains keywords such as event discount marketing, it is divided into text containing keywords such as event discount marketing (determining the text type as the marketing text), and the marketing activity list library (second preset database) is matched according to key information such as the user type and province (regional information) of the target user, and all marketing activities that meet the text to be processed are combined into a complete activity introduction (second reply text) according to the order of marketing activity name, applicable user type, identity, start time, end time, and activity rules, and lively mood particles (second mood particles) are selected from the happy mood particle library (first preset mood particle database) and added to the activity introduction, and finally the arranged activity introduction link is used as the target reply text.

[0197] It should be noted that before determining the target reply text, it is necessary to create a second preset database, for example: create a marketing campaign list library (second preset database), the library elements consist of {marketing campaign name, applicable user type, province, start time, end time, activity rules}, and it is updated in real time according to the timeliness of the activity.

[0198] In one possible implementation of the embodiment of the present disclosure, the above describes the process of determining the target reply text when the text type is the operation text and the marketing text. Next, the process of determining the target reply text when the text type is the other text is described. When the text type is the other text, the present disclosure provides a flow chart of another method for determining the target reply text, such as Figure 5 As shown, including:

[0199] Step 501, input the text to be processed into a preset text recognition model for text prediction processing to obtain a second reply candidate set corresponding to the text to be processed; wherein the second reply candidate set contains at least one third reply text.

[0200] In the embodiment of the present disclosure, the preset text recognition model is used to predict the text to be processed. The preset text recognition model is a custom selected model, such as a long short-term memory (LSTM) model, etc. The LSTM model is widely used in natural language processing (NLP), speech recognition, time series prediction and other fields. For example, in NLP, LSTM can be used for tasks such as sentiment analysis, machine translation, and text generation. Specifically, the embodiment of the present disclosure does not limit the preset text recognition model.

[0201] Step 502: Perform sentiment classification processing on each of the third reply texts in the second reply candidate set to obtain a second sentiment corresponding to each of the third reply texts.

[0202] In the disclosed embodiment, since a plurality of third reply texts, namely the second reply candidate set, are obtained when the text to be processed is predictively processed by the preset text recognition model, it is necessary to calculate the second emotion corresponding to each of the third reply texts so as to determine the target reply text corresponding to the text to be processed from the second reply candidate set according to the second emotion.

[0203] Regarding the sentiment classification of the third reply text, it can be performed through but not limited to: a BLSTM network and a CNN audio sentiment classifier.

[0204] Step 503, respectively calculating the emotion similarity between the first emotion and each of the second emotions to obtain emotion similarity values, and performing a comparison process on each of the emotion similarity values ​​to obtain the maximum emotion similarity value among all the emotion similarity values.

[0205] In the embodiment of the present disclosure, before calculating the emotion similarity value, it is also necessary to convert the first emotion and the second emotion into vectors, for example: perform data conversion processing on each of the second emotions respectively to obtain the first emotion vector corresponding to each of the second emotions; perform data conversion processing on the first emotion to obtain the second emotion vector corresponding to the first emotion; calculate the similarity between the first emotion vector and the second emotion vector through the preset similarity algorithm to obtain the third similarity value.

[0206] Regarding the calculation of the third similarity value, please refer to the description in the above step 104, so it will not be described in detail here.

[0207] Since it is necessary to determine the third reply text with the highest sentiment similarity to the text to be processed, it is necessary to compare each of the sentiment similarity values ​​to determine the third reply text with the highest sentiment similarity to the text to be processed according to the maximum sentiment similarity value.

[0208] Step 504: Use the third reply text corresponding to the maximum sentiment similarity value as the second initial reply text.

[0209] In the disclosed embodiment, the third reply text corresponding to the maximum emotion similarity value is the third reply text having the highest emotion similarity with the text to be processed, that is, the second initial reply text.

[0210] Step 505, obtain the regional information corresponding to the target user, and perform matching processing in the second preset modal particle database according to the regional information to obtain the third modal particle; wherein, the target user is the target who sends the voice to be processed and to whom a voice reply is to be made, and the second modal particle database is a pre-configured database containing multiple types of the third modal particles.

[0211] In the embodiment of the present disclosure, the geographical information may be the province information of the target user, or may be information divided according to the geographical location area based on the province information of the target user. Specifically, the embodiment of the present disclosure does not limit the geographical information.

[0212] The second preset modal particle database is a custom-set database, which contains a plurality of the third modal particles, and the third modal particles are custom-set modal particles, for example: modal particles with northern characteristics, modal particles with southern characteristics, modal particles with central characteristics, etc. Specifically, the present disclosed embodiment does not limit the third modal particles and the second preset modal particle database.

[0213] Step 506: merge the second initial reply text and the third modal particle to obtain the target reply text.

[0214] In the disclosed embodiment, corresponding emotion category modal particles (third modal particles) are selected according to the regional location (regional information) and added to the selected reply (second initial reply text) to obtain the target reply text, so that the target reply text has regional characteristics and can enhance the user's favorability.

[0215] In summary, when the text type is the other text, the process of determining the target reply text can be achieved by but not limited to the following methods: if no elements are matched in the sensitive word library, the standard operation manual answer library, and the event marketing keyword library, the text is classified into other text classes for processing (determining that the text type is the other text), the text information is input into the trained LSTM language model (preset text recognition model) to obtain a second reply candidate set, and the second reply candidate set is subjected to semantic sentiment classification using a semantic sentiment classifier, and one of the above five types of sentiment results, namely, the second sentiment, is also obtained; the first sentiment and the second sentiment are compared, and the third reply text with the highest similarity is obtained as the second initial reply text; according to the user's province information (regional information), according to the geographical location area division, the corresponding sentiment category modal particles are selected according to the regional location, the northern part selects modal particles with northern characteristics, the southern part selects modal particles with southern characteristics, and the central part selects modal particles with central characteristics, and adds them to the second initial reply text. At this time, the target reply text obtained has regional characteristics, which improves the user's favorability. Among them, the training set of the preset text recognition model is the Chinese Wikipedia corpus, and the language model of the reply message is trained to obtain a set of candidate reply sets.

[0216] In one implementable manner of the disclosed embodiment, the above describes the process of determining the target reply text when the text type is the operation text, marketing text, or other text. Next, the process of determining the target reply text when the text type is the sensitive text is described. When the text type is the sensitive text, the target reply text can be determined by but not limited to the following manner: extracting a fourth reply text from a third preset database as the target reply text, wherein the fourth reply text is a pre-set reply text, and the third preset database contains at least one fourth reply text.

[0217] In the embodiment of the present disclosure, the third preset database is a database of custom settings, and the fourth reply text is a text of custom settings, for example: "Sorry, I don't understand what you mean", etc. Specifically, the embodiment of the present disclosure does not limit the third preset database and the fourth reply text.

[0218] Therefore, for the text to be processed that is classified as sensitive word text, a safe reply can be directly used as the automatic reply text.

[0219] In one implementable manner of the embodiment of the present disclosure, as a refinement of the above-mentioned step 103, when performing speech synthesis processing, it can also be implemented in the following manner but not limited to: based on the preset speech synthesis model, the target reply text is subjected to text analysis processing to obtain the context attributes of the target reply text; the target reply text is subjected to clustering decision processing according to the context attributes and the clustering decision tree to obtain model sequence data; wherein, the preset speech synthesis model includes the clustering decision tree, and the clustering decision tree at least includes a fundamental frequency decision tree, a spectrum decision tree and a duration decision tree; parameter generation processing is performed according to the model sequence data to obtain fundamental frequency parameters, spectrum parameters and duration parameters corresponding to the target reply text; wherein, the model sequence data at least includes fundamental frequency data, spectrum data and duration data of the target reply text; speech synthesis processing is performed according to the fundamental frequency parameters, the spectrum parameters and the duration parameters to obtain the target reply speech.

[0220] In the embodiments of the present disclosure, for ease of understanding, an example is provided for illustration: during the synthesis process, the input text (target reply text) is analyzed to obtain context attributes, and decisions are made on the clustering decision trees of fundamental frequency, spectrum and duration according to the context attributes to obtain corresponding model sequence data, and the fundamental frequency and spectrum parameters are generated using the HMGenS tool, and the final speech to be compared (target reply speech) is synthesized through the STRAIGHT synthesizer (the above tools are all provided by HTK).

[0221] In one possible implementation of the embodiment of the present disclosure, as a refinement of the above step 101, when performing sentiment classification processing, the embodiment of the present disclosure provides a flow chart of sentiment classification, such as Figure 6 As shown, including:

[0222] Step 601, preprocessing the text to be processed to obtain a word vector corresponding to the text to be processed, and preprocessing the speech to be processed to obtain a speech frame corresponding to the speech to be processed.

[0223] In the embodiments of the present disclosure, the preprocessing of the text to be processed includes but is not limited to: using the Jieba word segmentation method to segment the text, and then using the Word2Vec model to obtain word vectors; the preprocessing of the speech to be processed includes but is not limited to: performing frame segmentation and windowing operations on the speech to be processed to obtain the speech frames.

[0224] Step 602: input the word vector into a preset semantic recognition model for semantic recognition processing to obtain semantic features of the text to be processed, and input the speech frame into a preset audio recognition model for audio recognition processing to obtain audio features of the speech to be processed.

[0225] In the embodiments of the present disclosure, the preset semantic recognition model is a model of custom settings, such as: BLSTM network model, etc., and the preset audio recognition model is a model of custom settings, such as: opensmile tool, etc. Specifically, the embodiments of the present disclosure do not limit the preset semantic recognition model and the preset audio recognition model.

[0226] Step 603: Input the semantic features into a preset sentiment classification model for sentiment classification processing to obtain the semantic sentiment of the text to be processed, and input the audio features into the preset sentiment classification model for sentiment classification processing to obtain the audio sentiment of the speech to be processed.

[0227] In the embodiments of the present disclosure, for ease of understanding, an example is provided for illustration: by sending semantic features into a trained CNN semantic sentiment classifier, semantic sentiment classification (speech sentiment) can be obtained, and by sending acoustic features into a trained CNN audio sentiment classifier, speech sentiment classification (audio sentiment) can be obtained.

[0228] Step 604: fuse the semantic emotion with the audio emotion to obtain the first emotion.

[0229] In the disclosed embodiment, since the semantic emotion and the audio emotion respectively contain multiple emotion types, it is necessary to calculate the score of each emotion through the decision layer fusion formula, and the highest score is taken as the final bimodal speech emotion classification result (first emotion), that is, the semantic emotion and the audio emotion need to be fused.

[0230] The fusion processing of the semantic emotion and the audio emotion can be implemented by, but not limited to, formula (3):

[0231] Score=w1*Score(T)+w2*Score(S) formula (3)

[0232] Among them, Score(T) represents the output result of semantic feature emotion recognition (semantic emotion), Score(S) represents the output result of audio feature emotion recognition (audio emotion), w1 represents the weight corresponding to the semantic feature, w2 represents the weight corresponding to the audio feature, and both w1 and w2 are determined by trial and error.

[0233] Regarding the acquisition of the first emotion, the embodiment of the present disclosure provides another flow chart of emotion classification, such as Figure 7 shown.

[0234] In summary, the acquisition of the first emotion can be achieved through but not limited to the following methods: using a classifier that integrates semantic and audio dual-modal speech emotion classification, after converting the processed speech into the processed text, preprocessing the processed text, inputting the preprocessed text into the BLSTM network to obtain semantic features, sending the semantic features into the trained CNN semantic emotion classifier to obtain semantic emotion classification (semantic emotion), using the OpenSmille tool to obtain the audio features of the speech, sending the audio features into the trained CNN audio emotion classifier to obtain the emotion classification of the speech (audio emotion), performing decision-making fusion on the semantic emotion classification results and the speech emotion classification results, calculating the score of each emotion through the decision-making layer fusion formula, and taking the highest score as the final dual-modal speech emotion classification result (first emotion).

[0235] In one possible implementation of the embodiment of the present disclosure, the preset speech synthesis model needs to be trained. Before the model training, in order to refine the probability of event occurrence and improve the accuracy of state annotation, the initial speech synthesis model needs to be improved to obtain the speech synthesis model to be trained. Regarding the acquisition of the speech synthesis model to be trained, the embodiment of the present disclosure provides a flow chart of a method for acquiring the speech synthesis model to be trained, as shown in FIG. Figure 8 As shown, including:

[0236] Step 801, modifying the dependency of the observation value parameters in the initial speech synthesis model according to the adjacent time state of the initial speech synthesis model to obtain a modified initial speech synthesis model; wherein the adjacent time state includes the model state of the initial speech synthesis model at the previous adjacent time of the current time, and the model state of the initial speech synthesis model at the next adjacent time of the current time.

[0237] In the embodiments of the present disclosure, in order to describe the dependency of the appearance of the observed values ​​of the two parameters, spectrum and fundamental frequency, on the annotations of their previous and subsequent states, the HMM model (initial speech synthesis model) is improved. Taking into account the continuity of speech, when improving the initial speech synthesis model, the appearance of the observed values ​​will be made dependent on the previous state and the next state, so as to obtain a more reasonable initial speech synthesis model.

[0238] At time t, the appearance of the observation value Oi depends on the state Si-1 at time t-1 and the state Si+1 at time t+1. At this time, the initial speech synthesis model needs to be modified to obtain a new emission probability (modified initial speech synthesis model). The modified initial speech synthesis model can be obtained by using but not limited to formula (4):

[0239] BB={b ij(k) =P(O t =V k |q t =S i-1 +S i+1 ,q t =S j )} Formula (4)

[0240] Wherein, V = {V1, V2, ..., VN}, is the symbol output set in the five-tuple description information set of the HMM model (initial speech synthesis model); S = {S1, S2, ..., SM}, is the state set in the five-tuple description information set; b ij(k) Indicates that in S o-1 ,S i+1 The probability of the symbol output set Vk appearing in the state.

[0241] Step 802, modifying the initial state probability matrix in the modified initial speech synthesis model according to the adjacent time states to obtain the speech synthesis model to be trained.

[0242] In the embodiment of the present disclosure, since the modified initial speech synthesis model changes the dependence of the observation value to the state at time t-1 and time t+1, in order to make the output state probability more accurate, it is necessary to use a logistic regression method to modify the initial state probability of the modified initial speech synthesis model based on the hidden state to obtain a more reasonable HMM model (speech synthesis model to be trained).

[0243] Since the hidden state z of the modified initial speech synthesis model t The space at time point t is {0,1}. That is, z t ∈{0,1} is a binary variable. Therefore, the speech synthesis model to be trained can be obtained by using, but not limited to, formula (5) and formula (6):

[0244] P(z t =1) = π t , formula (5)

[0245]

[0246] Where t = 1,…,T; π is the initial state probability matrix; β is the random coefficient, β∈{0,1}; p(z t =1) =π t represents the probability matrix of the implicit state at the initial time t = 1. Thus, the modified initial state probability π can be obtained t .

[0247] Compared with the traditional HMM (initial speech synthesis model), the output probability of the observation value depends only on the current state of the model. The output probability of the observation value at time t of the improved HMM (speech synthesis model to be trained) depends on the state at time t and t+1. Taking spectrum extraction as an example, there is a sample to be extracted. After block division, the observation value sequence "spectral segment a|spectral segment b|spectral segment c|spectral segment d" is obtained, where O2 = spectrum segment b, the state is medium and low frequency, assuming that O1 has been marked as medium and high frequency, the analysis of the improved HMM model and the HMM model in the ability to identify spectrum parameters is as follows:

[0248] Traditional HMM model (initial speech synthesis model):

[0249] P(q2=low frequency)=P(q2=low frequency|q1=medium and high frequency)P(O2|q2=low frequency)

[0250] P(q2=mid-low frequency)=P(q2=mid-low frequency|q1=mid-high frequency)P(O2|q2=mid-low frequency)

[0251] From the state transition probability, we can see that the probability of O2 being output as a low-frequency or medium-low-frequency state is basically equal, and it is wrong for O2 to be output as a low-frequency state.

[0252] Improved HMM model (speech synthesis model to be trained):

[0253] P(q2=low frequency)=P(q2=low frequency|q1=medium and high frequency)*P(O2|q2=low frequency,q3=high frequency)

[0254] P(q2=mid-low frequency)=P(q2=mid-low frequency|q1=mid-high frequency)*P(O2|q2=mid-low frequency,q3=high frequency)

[0255] According to the transfer matrix, the probability of low frequency and high frequency appearing as continuous states approaches zero, P(q2=low frequency)=0, so the state of O2 is medium-low frequency.

[0256] It can be seen from the above that the improved HMM model (speech synthesis model to be trained) can refine the probability of event occurrence and make the state labeling more accurate.

[0257] In one implementable method of the embodiment of the present disclosure, it is necessary to obtain the preset speech synthesis model after performing model training processing on the speech synthesis model to be trained, so as to realize the synthesis processing of the target reply speech through the preset speech synthesis model. Regarding the training of the speech synthesis model to be trained, it can be implemented in but not limited to the following methods: obtaining a first preset number of first training data and training context attributes; wherein the first training data at least includes training acoustic parameters, and the training context attributes are context attributes that have an impact on the training acoustic parameters; performing model training processing on the speech synthesis model to be trained through the first training data to obtain the single factor state of the speech synthesis model to be trained, and The speech synthesis model to be trained is subjected to model training processing according to context attributes to obtain the context state of the speech synthesis model to be trained; the speech synthesis model to be trained is subjected to decision clustering processing according to a preset decision tree clustering algorithm to obtain the clustering state of the speech synthesis model to be trained, and the first residence time of the single factor state, the second residence time of the context state and the third residence time of the clustering state are obtained; the preset duration model is initialized according to the first residence time, the second residence time and the third residence time to obtain the initial duration model; the speech synthesis model to be trained and the initial duration model are subjected to model clustering processing according to the preset decision tree clustering algorithm to obtain the preset speech synthesis model.

[0258] In the embodiment of the present disclosure, in order to facilitate understanding of the implementation process of the embodiment of the present disclosure, a schematic diagram of the training process of a preset speech synthesis model is provided, such as Fig. 9 As shown, the parameter prediction method needs to model the spectrum, duration parameters and fundamental frequency parameters, and produce their own state decision trees at the same time. The parameter prediction method is to first use the decision tree to predict the model, and then generate parameters according to the dynamic characteristics. Modeling parameter configuration: The system mainly processes Mandarin, using 24-order MCEP parameters and 27-dimensional parameters of first-order energy and second-order fundamental frequency. There are 66 basic modeling units, including 23 initials, 24 finals, 16 overall recognition, and 3 medials. The improved HMM model with no gaps from left to right is adopted. In terms of data preparation, STRAIGHT is used to analyze wav data to obtain acoustic data (spectrum and fundamental frequency), and the annotation includes rhythmic annotation. In addition, some contextual attributes that affect acoustic parameters (fundamental frequency, duration, MEL spectrum) are selected based on prior knowledge, and corresponding question sets are designed for decision tree clustering.

[0259] Use Hinit and Hcompv to initialize the monophone model. The Hinit tool uses the Viterbi algorithm to initialize the HMM model through the time axis of the training data. The Hcompv tool performs flat initialization on the model, calculates the global variance and estimates the lower limit of the variance. The VFLoors file will be output, containing the global change vector multiplied by the factor. The value stored in varFloor1 is used as the basis of the estimated change vector in the subsequent training process. In the training iteration, the number of training frames associated with a specific HMM model state may be very low, and the estimated change value of the state will be very small. In this case, the basis can be used instead to avoid the change value tending to the performance or even causing calculation errors. Through multiple HRest iterations, the improved HMM model parameters (main mean, variance, etc.) are updated. The change metric is used to indicate convergence at each HRest iteration. When this metric value no longer decreases from one HRest iteration to the next, the HRest iteration process ends. Finally, a single-factor HMM model file is obtained. Next, the model is expanded according to the context problem set, and the context-related model file is obtained by embedded training using the HERest tool. Finally, the context-dependent model is clustered using a decision tree-based clustering method to improve the robustness of the model and the balance between the data volumes. The clustered model is then further trained to optimize the model parameters and obtain the statistical data of the state residence time of each state model. This data is then used to initialize the duration model, and the HHEd tool is used to cluster the model using a decision tree method. The trained model includes the spectrum, fundamental frequency and duration parameter clustering HMM and the decision tree of each state.

[0260] In one implementable manner of the embodiment of the present disclosure, when the first similarity value is less than or equal to the preset similarity threshold, it indicates that the accuracy of the preset speech synthesis model is low. Therefore, the preset speech synthesis model needs to be updated to obtain a more accurate speech synthesis model. Therefore, after obtaining the first similarity value, the following method can be adopted but is not limited to: when the first similarity value is less than or equal to the preset similarity threshold, obtaining a second preset amount of second training data; performing model training processing on the preset speech synthesis model according to the reply voice text and the second training data to obtain an updated speech synthesis model.

[0261] In the disclosed embodiment, ASR is used to convert synthesized speech (target reply speech) into new text (speech reply text), and the cosine similarity of the speech reply text and the text to be processed is calculated. The threshold is set to 0.9. If it is less than 0.9, it indicates that the speech synthesis is unqualified, and the relevant parameters of the dynamic adjustment parameter prediction model are returned to adjust the context set, and the model is trained again to obtain parameters and perform speech synthesis. The above operations are repeated until the cosine similarity reaches the threshold, and the synthesized speech obtained in this case is used as the final automatic speech reply result (final reply speech).

[0262] In summary, in order to facilitate understanding of the voice reply method described in the present disclosure, the present disclosure provides a flow chart of another voice reply method, such as Fig.10 As shown, the overall process of the voice response method is included.

[0263] In summary, the embodiments of the present disclosure can achieve the following effects:

[0264] 1. The disclosed embodiment converts the received speech to be processed into text data (text to be processed), classifies the text data, determines the text emotion (first emotion) and text type of the text data, obtains reply data (target reply text) corresponding to different types of text data and text emotions, replies to the target reply text with speech (target reply speech), and then converts it into text (reply speech text), compares the reply speech text with the initial target reply text, and when the two texts have a high degree of similarity, uses the target reply speech as the final speech for reply, thereby realizing intelligent automatic reply without human intervention, improving the accuracy and degree of anthropomorphism, and enhancing the user experience.

[0265] 2. The disclosed embodiments focus on the analysis and processing of text information in speech. In text classification, in the processing of the second classification, an operation manual answer library is created, and the keywords of the extracted input text are matched with the operation manual answer library. If a unique keyword element object is matched, the answer in the corresponding element object is returned. If multiple element objects are matched, the cosine similarity of the input text and the questions of the matched element objects is calculated, and the answer to the question element with high similarity is taken as the answer. On the other hand, the semantic sentiment classification and speech sentiment classification in the speech sentiment classification process are interspersed into the text classification processing according to the weights. The sentiment classifier performs sentiment classification on the input text, and the sentiment classifier performs sentiment classification on the original speech of the text. According to the result of sentiment classification, modal particles are added to the obtained answer, and finally converted into speech information using a dynamic tts model based on HMM as the final automatic voice reply.

[0266] 3. The dynamic tts model based on the improved HMM used in the embodiment of the present disclosure has an observation value output probability at time t that depends on the state at time t and time t+1, which can refine the probability of an event occurring and make the state labeling more accurate.

[0267] 4. The disclosed embodiments make full use of emotional factors, regional factors, user type factors, etc. to make voice replies more friendly and improve user experience.

[0268] Corresponding to the above-mentioned voice reply method, the present invention also provides a voice reply device. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment, and will not be repeated in the present invention.

[0269] Fig.11 A schematic diagram of the structure of a voice reply device provided in an embodiment of the present disclosure, such as Fig.11 As shown, including:

[0270] The conversion unit 1101 is used to perform speech conversion processing on the received speech to be processed to obtain a text to be processed;

[0271] A first classification unit 1102 is used to perform emotion classification processing according to the speech to be processed and the text to be processed to obtain a first emotion corresponding to the speech to be processed;

[0272] The second classification unit 1103 is used to perform text classification processing on the text to be processed to obtain a text type corresponding to the text to be processed;

[0273] A first determining unit 1104 is used to determine a target reply text corresponding to the to-be-processed text according to the text type and the first emotion; wherein the text type includes sensitive text, operation text, marketing text, and other text;

[0274] The synthesis unit 1105 is used to input the target reply text into a preset speech synthesis model for speech synthesis processing to obtain a target reply speech;

[0275] The recognition unit 1106 is used to perform speech recognition processing on the target reply speech to obtain a reply speech text corresponding to the target reply speech;

[0276] The calculation unit 1107 is used to calculate the similarity between the reply voice text and the text to be processed to obtain a first similarity value;

[0277] The second determining unit 1108 is configured to use the target reply speech as the final reply speech corresponding to the speech to be processed when the first similarity value is greater than a preset similarity threshold.

[0278] The device for voice reply provided by the present disclosure performs voice conversion processing on the received voice to be processed to obtain a text to be processed, and performs emotion classification processing on the voice to be processed and the text to be processed to obtain a first emotion corresponding to the voice to be processed; performs text classification processing on the text to be processed to obtain a text type corresponding to the text to be processed, and determines a target reply text corresponding to the text to be processed according to the text type and the first emotion; wherein the text type includes sensitive text, operation text, marketing text, and other text; inputs the target reply text into a preset voice synthesis model for voice synthesis processing to obtain a target reply voice, and performs voice recognition processing on the target reply voice to obtain a reply voice text corresponding to the target reply voice; calculates the similarity between the reply voice text and the text to be processed to obtain a first similarity value, and when the first similarity value is greater than a preset similarity threshold, uses the target reply voice as the final reply voice corresponding to the voice to be processed. Compared with the related art, the embodiment of the present disclosure converts the received voice to be processed into text data (text to be processed), classifies the text data, determines the text emotion (first emotion) and text type of the text data, obtains reply data (target reply text) corresponding to different types of text data and text emotions, replies to the target reply text with voice (target reply voice), and then converts it into text (reply voice text), compares the reply voice text with the initial target reply text, and when the two texts have a high degree of similarity, uses the target reply voice as the final voice for reply, thereby realizing intelligent automatic reply without human intervention, improving the accuracy and degree of anthropomorphism, and enhancing the user experience.

[0279] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Fig.12 As shown, the first determining unit 1104 includes:

[0280] The acquisition module 11041 is used to acquire the first type of information of the target user; wherein the target user is the target who sends the voice to be processed and to whom a voice reply is to be made;

[0281] A matching module 11042 is configured to perform matching processing in a first preset database according to the first type of information and the text to be processed to obtain a first reply candidate set; wherein the first reply candidate set contains at least one first reply text;

[0282] A calculation module 11043 is used to calculate the similarity between the to-be-processed text and each of the first reply texts in the first reply candidate set respectively to obtain a second similarity value when the number of the first reply texts in the first reply candidate set is greater than a preset number threshold;

[0283] The comparison module 11044 is used to compare each of the second similarity values ​​to obtain a maximum similarity value among all the second similarity values, and use the first reply text corresponding to the maximum similarity value as the first initial reply text;

[0284] The matching module 11042 is further used to perform matching processing from different preset emotion databases according to the first emotion to obtain a target emotion database, and perform matching processing in the target emotion database according to the first initial reply text to obtain a first modal particle; wherein different preset emotion databases correspond to different emotion categories;

[0285] The merging module 11045 is used to merge the first initial reply text with the first modal particle to obtain the target reply text.

[0286] Furthermore, in a possible implementation of the embodiment of the present disclosure, the matching module 11042 is further configured to:

[0287] In the case where it is determined according to the first type of information that the target user is a new user, matching the to-be-processed text in the first operation database to obtain the first reply candidate set; wherein the target users are at least divided into the new users and the old users;

[0288] When it is determined that the target user is an old user based on the first type of information, the text to be processed is matched in the second operation database to obtain the first reply candidate set, wherein the first preset database includes the first operation database and the second operation database, and the data in the first operation database is different from the data in the second operation database.

[0289] Further, in a possible implementation of the embodiment of the present disclosure, the first determining unit 1104 is further configured to:

[0290] Acquire the user type of the target user and the region information corresponding to the target user; wherein the target user is the target who sends the voice to be processed and to whom a voice reply is to be made;

[0291] Perform matching processing in a second preset database according to the user type, the regional information and the text to be processed, and obtain a second reply text corresponding to the text to be processed;

[0292] According to the second reply text, matching processing is performed in a first preset modal particle database to obtain a second modal particle; wherein the first modal particle database is a pre-configured database containing a plurality of the second modal particles;

[0293] The second reply text and the second modal particle are combined to obtain the target reply text.

[0294] Further, in a possible implementation of the embodiment of the present disclosure, the first determining unit 1104 is further configured to:

[0295] Inputting the text to be processed into a preset text recognition model for text prediction processing to obtain a second reply candidate set corresponding to the text to be processed; wherein the second reply candidate set contains at least one third reply text;

[0296] Performing sentiment classification processing on each of the third reply texts in the second reply candidate set respectively to obtain a second sentiment corresponding to each of the third reply texts;

[0297] Calculating the emotion similarity between the first emotion and each of the second emotions respectively to obtain emotion similarity values, and performing comparison processing on each of the emotion similarity values ​​to obtain the maximum emotion similarity value among all the emotion similarity values;

[0298] Using the third reply text corresponding to the maximum sentiment similarity value as the second initial reply text;

[0299] Acquire the region information corresponding to the target user, and perform matching processing in the second preset modal particle database according to the region information to obtain the third modal particle; wherein the target user is the target who sends the voice to be processed and to be responded to by voice, and the second modal particle database is a pre-configured database containing a plurality of the third modal particles;

[0300] The second initial reply text and the third modal particle are combined to obtain the target reply text.

[0301] Further, in a possible implementation of the embodiment of the present disclosure, the first determining unit 1104 is further configured to:

[0302] A fourth reply text is extracted from a third preset database as the target reply text, wherein the fourth reply text is a preset reply text, and the third preset database contains at least one fourth reply text.

[0303] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Fig.12 As shown, the synthesis unit 1105 includes:

[0304] The analysis module 11051 is used to perform text analysis on the target reply text based on the preset speech synthesis model to obtain context attributes of the target reply text;

[0305] A decision module 11052 is used to perform clustering decision processing on the target reply text according to the context attributes and the clustering decision tree to obtain model sequence data; wherein the preset speech synthesis model includes the clustering decision tree, and the clustering decision tree at least includes a fundamental frequency decision tree, a spectrum decision tree and a duration decision tree;

[0306] A generating module 11053 is used to perform parameter generation processing according to the model sequence data to obtain fundamental frequency parameters, spectrum parameters and duration parameters corresponding to the target reply text; wherein the model sequence data at least includes fundamental frequency data, spectrum data and duration data of the target reply text;

[0307] The synthesis module 11054 is used to perform speech synthesis processing according to the fundamental frequency parameters, the spectrum parameters and the duration parameters to obtain the target reply speech.

[0308] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Fig.12 As shown, the first classification unit 1102 includes:

[0309] The processing module 11021 is used to perform a preprocessing operation on the text to be processed to obtain a word vector corresponding to the text to be processed, and to perform a preprocessing operation on the speech to be processed to obtain a speech frame corresponding to the speech to be processed;

[0310] The recognition module 11022 is used to input the word vector into a preset semantic recognition model for semantic recognition processing to obtain semantic features of the text to be processed, and input the speech frame into a preset audio recognition model for audio recognition processing to obtain audio features of the speech to be processed;

[0311] The classification module 11023 is used to input the semantic features into a preset emotion classification model for emotion classification processing to obtain the semantic emotion of the text to be processed, and input the audio features into the preset emotion classification model for emotion classification processing to obtain the audio emotion of the speech to be processed;

[0312] The fusion module 11024 is used to fuse the semantic emotion with the audio emotion to obtain the first emotion.

[0313] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Fig.12 As shown, the device also includes:

[0314] A modification unit 1109, configured to modify the dependency of the observation value parameters in the initial speech synthesis model according to the adjacent time state of the initial speech synthesis model, so as to obtain a modified initial speech synthesis model; wherein the adjacent time state includes the model state of the initial speech synthesis model at the previous adjacent time of the current time, and the model state of the initial speech synthesis model at the next adjacent time of the current time;

[0315] The modification unit 1109 is further used to modify the initial state probability matrix in the modified initial speech synthesis model according to the adjacent time state to obtain the speech synthesis model to be trained.

[0316] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Fig.12 As shown, the device also includes:

[0317] The acquisition unit 1110 is used to acquire a first preset amount of first training data and training context attributes; wherein the first training data at least includes training acoustic parameters, and the training context attributes are context attributes that have an impact on the training acoustic parameters;

[0318] The training unit 1111 is used to perform model training processing on the speech synthesis model to be trained by using the first training data to obtain the single factor state of the speech synthesis model to be trained, and to perform model training processing on the speech synthesis model to be trained by using the training context attributes to obtain the context state of the speech synthesis model to be trained;

[0319] The training unit 1111 is also used to perform decision clustering processing on the speech synthesis model to be trained by a preset decision tree clustering algorithm to obtain a clustering state of the speech synthesis model to be trained;

[0320] The acquisition unit 1110 is further configured to acquire a first stay time of the single factor state, a second stay time of the context state, and a third stay time of the cluster state;

[0321] The processing unit 1112 is used to initialize the preset duration model according to the first residence time, the second residence time and the third residence time to obtain an initial duration model;

[0322] The training unit 1111 is also used to perform model clustering processing on the speech synthesis model to be trained and the initial duration model through the preset decision tree clustering algorithm to obtain the preset speech synthesis model.

[0323] Further, in a possible implementation of the embodiment of the present disclosure, the acquisition unit 1110 is further configured to, when the first similarity value is less than or equal to the preset similarity threshold, acquire a second preset amount of second training data;

[0324] The training unit 1111 is also used to perform model training processing on the preset speech synthesis model according to the reply voice text and the second training data to obtain an updated speech synthesis model.

[0325] It should be noted that the above explanation of the method embodiment is also applicable to the device of the embodiment of the present disclosure, and the principle is the same, which is no longer limited in the embodiment of the present disclosure.

[0326] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0327] Fig.13 A schematic block diagram of an example electronic device 1300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0328] like Fig.13 As shown, the device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 1302 or a computer program loaded from a storage unit 1308 to a RAM (Random Access Memory) 1303. In the RAM 1303, various programs and data required for the operation of the device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An I / O (Input / Output) interface 1305 is also connected to the bus 1304.

[0329] A number of components in the device 1300 are connected to the I / O interface 1305, including: an input unit 1306, such as a keyboard, a mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a disk, an optical disk, etc.; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0330] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 1301 performs the various methods and processes described above, such as the method of voice reply. For example, in some embodiments, the method of voice reply may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to execute the aforementioned voice reply method in any other appropriate manner (eg, by means of firmware).

[0331] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a special purpose or general purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0332] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0333] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an EPROM (Electrically Programmable Read-Only-Memory) or a flash memory, an optical fiber, a CD-ROM (Compact Dis sc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0334] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0335] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0336] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0337] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0338] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0339] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A voice reply method, characterized in that: include: Performing voice conversion processing on the received speech to be processed to obtain a text to be processed, and performing emotion classification processing on the speech to be processed and the text to be processed to obtain a first emotion corresponding to the speech to be processed; Performing text classification processing on the text to be processed to obtain a text type corresponding to the text to be processed, and determining a target reply text corresponding to the text to be processed according to the text type and the first emotion; wherein the text type includes sensitive text, operation text, marketing text, and other text; Inputting the target reply text into a preset speech synthesis model for speech synthesis processing to obtain a target reply speech, and performing speech recognition processing on the target reply speech to obtain a reply speech text corresponding to the target reply speech; The similarity between the reply voice text and the text to be processed is calculated to obtain a first similarity value, and when the first similarity value is greater than a preset similarity threshold, the target reply voice is used as the final reply voice corresponding to the voice to be processed.

2. The method according to claim 1, characterized in that: If the text type is the operation text, then determining the target reply text corresponding to the to-be-processed text according to the text type and the first emotion includes: Acquire first type information of a target user; wherein the target user is a target who sends the voice to be processed and to whom a voice reply is to be made; Performing matching processing in a first preset database according to the first type of information and the text to be processed, obtaining a first reply candidate set; wherein the first reply candidate set includes at least one first reply text; When the number of the first reply texts in the first reply candidate set is greater than a preset number threshold, respectively calculating the similarity between the to-be-processed text and each of the first reply texts in the first reply candidate set to obtain a second similarity value; Compare each of the second similarity values ​​to obtain a maximum similarity value among all the second similarity values, and use the first reply text corresponding to the maximum similarity value as the first initial reply text; According to the first emotion, matching processing is performed from different preset emotion databases to obtain a target emotion database, and according to the first initial reply text, matching processing is performed in the target emotion database to obtain a first modal particle; wherein different preset emotion databases correspond to different emotion categories; The first initial reply text and the first modal particle are combined to obtain the target reply text.

3. The method according to claim 2, characterized in that The matching process is performed in a first preset database according to the first type of information and the text to be processed to obtain a first reply candidate set, which includes: In the case where it is determined according to the first type of information that the target user is a new user, matching the to-be-processed text in the first operation database to obtain the first reply candidate set; wherein the target users are at least divided into the new users and the old users; When it is determined that the target user is an old user based on the first type of information, the text to be processed is matched in the second operation database to obtain the first reply candidate set, wherein the first preset database includes the first operation database and the second operation database, and the data in the first operation database is different from the data in the second operation database.

4. The method according to claim 1, characterized in that: If the text type is the marketing text, determining the target reply text corresponding to the to-be-processed text according to the text type and the first emotion includes: Acquire the user type of the target user and the region information corresponding to the target user; wherein the target user is the target who sends the voice to be processed and to whom a voice reply is to be made; Perform matching processing in a second preset database according to the user type, the regional information and the text to be processed, and obtain a second reply text corresponding to the text to be processed; According to the second reply text, matching processing is performed in a first preset modal particle database to obtain a second modal particle; wherein the first modal particle database is a pre-configured database containing a plurality of the second modal particles; The second reply text and the second modal particle are combined to obtain the target reply text.

5. The method according to claim 1, characterized in that If the text type is the other text, then determining the target reply text corresponding to the to-be-processed text according to the text type and the first emotion includes: Inputting the text to be processed into a preset text recognition model for text prediction processing to obtain a second reply candidate set corresponding to the text to be processed; wherein the second reply candidate set contains at least one third reply text; Performing sentiment classification processing on each of the third reply texts in the second reply candidate set respectively to obtain a second sentiment corresponding to each of the third reply texts; Calculating the emotion similarity between the first emotion and each of the second emotions respectively to obtain emotion similarity values, and performing comparison processing on each of the emotion similarity values ​​to obtain the maximum emotion similarity value among all the emotion similarity values; Using the third reply text corresponding to the maximum sentiment similarity value as the second initial reply text; Acquire the region information corresponding to the target user, and perform matching processing in the second preset modal particle database according to the region information to obtain the third modal particle; wherein the target user is the target who sends the voice to be processed and to be responded to by voice, and the second modal particle database is a pre-configured database containing a plurality of the third modal particles; The second initial reply text and the third modal particle are combined to obtain the target reply text.

6. The method according to claim 1, characterized in that If the text type is the sensitive text, determining the target reply text corresponding to the to-be-processed text according to the text type and the first emotion includes: A fourth reply text is extracted from a third preset database as the target reply text, wherein the fourth reply text is a preset reply text, and the third preset database contains at least one fourth reply text.

7. The method according to claim 1, characterized in that The step of inputting the target reply text into a preset speech synthesis model for speech synthesis processing to obtain the target reply speech comprises: Based on the preset speech synthesis model, performing text analysis processing on the target reply text to obtain context attributes of the target reply text; The target reply text is clustered and decided according to the context attributes and the clustering decision tree to obtain model sequence data; wherein the preset speech synthesis model includes the clustering decision tree, and the clustering decision tree includes at least a fundamental frequency decision tree, a spectrum decision tree and a duration decision tree; Performing parameter generation processing according to the model sequence data to obtain fundamental frequency parameters, spectrum parameters and duration parameters corresponding to the target reply text; wherein the model sequence data at least includes fundamental frequency data, spectrum data and duration data of the target reply text; Speech synthesis processing is performed according to the fundamental frequency parameter, the spectrum parameter and the duration parameter to obtain the target reply speech.

8. The method according to claim 1, characterized in that The performing emotion classification processing according to the speech to be processed and the text to be processed to obtain the first emotion corresponding to the speech to be processed comprises: Preprocessing the text to be processed to obtain a word vector corresponding to the text to be processed, and preprocessing the speech to be processed to obtain a speech frame corresponding to the speech to be processed; Inputting the word vector into a preset semantic recognition model for semantic recognition processing to obtain semantic features of the text to be processed, and inputting the speech frame into a preset audio recognition model for audio recognition processing to obtain audio features of the speech to be processed; Inputting the semantic features into a preset emotion classification model for emotion classification processing to obtain the semantic emotion of the text to be processed, and inputting the audio features into the preset emotion classification model for emotion classification processing to obtain the audio emotion of the speech to be processed; The semantic emotion is fused with the audio emotion to obtain the first emotion.

9. The method according to claim 1, characterized in that: Before performing voice conversion processing on the received speech to be processed to obtain the text to be processed, the method further includes: Modifying the dependency of the observation value parameters in the initial speech synthesis model according to the adjacent time state of the initial speech synthesis model to obtain a modified initial speech synthesis model; wherein the adjacent time state includes the model state of the initial speech synthesis model at the previous adjacent time of the current time, and the model state of the initial speech synthesis model at the next adjacent time of the current time; The initial state probability matrix in the modified initial speech synthesis model is modified according to the adjacent time states to obtain the speech synthesis model to be trained.

10. The method according to claim 9, characterized in that After modifying the initial state probability matrix in the modified initial speech synthesis model according to the adjacent time states to obtain the speech synthesis model to be trained, the method further includes: Acquire a first preset amount of first training data and training context attributes; wherein the first training data at least includes training acoustic parameters, and the training context attributes are context attributes that have an impact on the training acoustic parameters; Performing model training processing on the speech synthesis model to be trained using the first training data to obtain a single factor state of the speech synthesis model to be trained, and performing model training processing on the speech synthesis model to be trained using the training context attributes to obtain a context state of the speech synthesis model to be trained; Performing decision clustering processing on the speech synthesis model to be trained by a preset decision tree clustering algorithm to obtain the clustering state of the speech synthesis model to be trained, and obtaining the first residence time of the single factor state, the second residence time of the context state, and the third residence time of the cluster state; Initializing a preset duration model according to the first residence time, the second residence time, and the third residence time to obtain an initial duration model; The preset decision tree clustering algorithm is used to perform model clustering processing on the speech synthesis model to be trained and the initial duration model to obtain the preset speech synthesis model.

11. The method according to claim 1, characterized in that After calculating the similarity between the reply voice text and the text to be processed to obtain a first similarity value, the method further includes: When the first similarity value is less than or equal to the preset similarity threshold, obtaining a second preset amount of second training data; The preset speech synthesis model is subjected to model training processing according to the reply speech text and the second training data to obtain an updated speech synthesis model.

12. A voice reply device, characterized in that: include: A conversion unit, used for performing speech conversion processing on the received speech to be processed to obtain a text to be processed; A first classification unit is used to perform emotion classification processing according to the speech to be processed and the text to be processed to obtain a first emotion corresponding to the speech to be processed; A second classification unit is used to perform text classification processing on the text to be processed to obtain a text type corresponding to the text to be processed; A first determination unit is used to determine a target reply text corresponding to the to-be-processed text according to the text type and the first emotion; wherein the text type includes sensitive text, operation text, marketing text, and other text; A synthesis unit, used for inputting the target reply text into a preset speech synthesis model for speech synthesis processing to obtain a target reply speech; A recognition unit, configured to perform speech recognition processing on the target reply speech to obtain a reply speech text corresponding to the target reply speech; A calculation unit, used for calculating the similarity between the reply voice text and the text to be processed to obtain a first similarity value; The second determining unit is used to use the target reply voice as the final reply voice corresponding to the to-be-processed voice when the first similarity value is greater than a preset similarity threshold.

13. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.

15. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images