Speech generation method, device, equipment and storage medium

Abstract

The present application relates to a speech generation method, device, equipment and storage medium. The method includes: identifying the state of the first user based on the image of the first user, and obtaining the state characteristics of the first user; the image includes image content for presenting the first user; identifying the emotion type of the first user based on the state characteristics, and determining the response emotion type matching the emotion type; obtaining the response sound characteristics and response emotion characteristics of the second user corresponding to the response emotion type; obtaining the response text content matching the state characteristics, and extracting the content characteristics of the response text content; fusing the content characteristics, the response emotion characteristics and the response sound characteristics to obtain the fusion characteristics; performing speech synthesis based on the fusion characteristics to generate the response speech of the second user under the response emotion type and used to express the response text content. This solution can reduce the waste of computer resources.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more particularly to the field of speech processing technology, and in particular to a speech generation method, apparatus, device and storage medium. Background Art

[0002] With the rapid development of science and technology, various advanced technologies have emerged, and speech processing technology is one of the important advanced technologies. Using speech processing technology to give machines speech output functions is currently a very popular use scenario. For example, home care robots can speak and communicate with users.

[0003] Then, in the traditional method, when the machine automatically outputs speech, it mainly outputs speech according to the preset sound, which is not flexible enough and has certain limitations, resulting in inaccurate output speech and often no effect. Therefore, generating inaccurate and invalid speech is a waste of computer resources to a certain extent. Moreover, since the generated speech is not accurate enough, it usually requires multiple speech outputs, which also wastes computer resources. Summary of the invention

[0004] Based on this, it is necessary to provide a speech generation method, apparatus, computer device and storage medium that can reduce the waste of computer resources in response to the above technical problems.

[0005] A speech generation method, the method comprising:

[0006] recognizing the state of the first user based on the image of the first user, and obtaining the state characteristics of the first user; the image includes image content for presenting the first user;

[0007] Identifying the emotion type of the first user based on the state feature, and determining a coping emotion type matching the emotion type;

[0008] Acquiring a response sound feature of a second user corresponding to the response emotion type;

[0009] Acquire a coping emotion feature corresponding to the coping emotion type;

[0010] Acquire the response text content that matches the state feature, and extract the content feature of the response text content;

[0011] fusing the content feature, the response emotion feature, and the response sound feature to obtain a fusion feature;

[0012] Speech synthesis is performed based on the fusion features to generate a response speech of the second user under the response emotion type and used to express the response text content.

[0013] A speech generating device, comprising:

[0014] a state recognition module, configured to recognize the state of the first user based on an image of the first user, and obtain state characteristics of the first user; the image includes image content for presenting the first user;

[0015] a coping emotion determination module, configured to identify the emotion type of the first user based on the state feature, and determine a coping emotion type matching the emotion type;

[0016] A feature acquisition module, used to acquire a response sound feature of the second user corresponding to the response emotion type; acquire a response emotion feature corresponding to the response emotion type; acquire a response text content matching the state feature, and extract a content feature of the response text content;

[0017] A feature fusion module, used to fuse the content feature, the response emotion feature and the response sound feature to obtain a fusion feature;

[0018] A speech synthesis module is used to perform speech synthesis based on the fusion feature to generate a response speech of the second user under the response emotion type and used to express the response text content.

[0019] In one of the embodiments, the coping emotion type is determined from the emotion library; the emotion library pre-stores the coping relationship between the emotion types; the coping emotion determination module is also used to search the emotion library for the emotion type of the second user used to cope with the emotion type of the first user based on the coping relationship, and obtain the coping emotion type.

[0020] In one embodiment, the state characteristics include the facial expression characteristics of the first user; the coping emotion determination module is also used to determine multiple candidate emotion types for coping with the first user's emotion type; obtain the facial expression characteristics of the second user under each of the candidate emotion types; match the facial expression characteristics of the first user with the facial expression characteristics of the second user under each candidate emotion type respectively; and determine the candidate emotion type to which the matched facial expression characteristics of the second user belong as the coping emotion type.

[0021] In one of the embodiments, the candidate emotion type is determined from the emotion types of the second user pre-stored in an emotion library; the emotion library also stores the corresponding second user's expression features and sound features for each emotion type of the second user; the feature acquisition module is also used to obtain the corresponding second user's expression features stored for each candidate emotion type from the emotion library; and obtain the corresponding second user's sound features stored for the response emotion type from the emotion library to obtain the response sound features.

[0022] In one embodiment, the device further comprises:

[0023] A feature extraction module is used to obtain sample speech of the second user under different emotion types; iteratively train a speech feature extraction model based on the sample speech of the second user under each emotion type, so that the speech feature extraction model iteratively learns the sound characteristic information of the second user under each emotion type from the sample speech, so as to iteratively update the sound feature coding information of the second user under the corresponding emotion type based on the sound characteristic information; obtain the sound feature coding information of the second user under each emotion type after the training is completed, and obtain the sound characteristics of the second user under each emotion type.

[0024] In one embodiment, the state characteristics include expression characteristics and voice characteristics of the first user; the state recognition module is also used to obtain a captured image of the first user; obtain voice data of the first user when the image is captured; identify the expression state of the first user from the image to obtain the expression characteristics of the first user; identify the voice state of the first user from the voice data to obtain the voice characteristics of the first user.

[0025] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in each embodiment of the present application when executing the computer program.

[0026] A computer-readable storage medium stores a computer program, which implements the steps in each embodiment of the present application when executed by a processor.

[0027] A computer program product includes a computer program, and when the computer program is executed by a processor, the steps in each embodiment of the present application are implemented.

[0028] The above-mentioned speech generation method, device, computer equipment, storage medium and computer program product identify the state of the first user based on the image of the first user, and identify the emotion type of the first user based on the identified state characteristics. Then, determine the response emotion type that matches the emotion type, obtain the response sound characteristics and response emotion characteristics of the second user corresponding to the response emotion type, and determine the matching response text content according to the state characteristics of the first user. The content characteristics, response emotion characteristics and response sound characteristics of the text content are integrated, and speech synthesis is performed based on the integrated fusion characteristics, so that a response speech can be generated that responds to the text content with the voice expression of the second user under the response emotion type. Compared with the traditional preset content with a fixed preset sound, a response speech can be adaptively generated to more accurately respond to the current state of the first user, which greatly improves flexibility, avoids limitations, and reduces the waste of computer resources to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A diagram showing an application environment of a speech generation method in an embodiment;

[0030] Figure 2 is a flow chart of a speech generation method in one embodiment;

[0031] Figure 3 is a simplified diagram of a speech generation method in one embodiment;

[0032] Figure 4 A schematic diagram of the principle of a speech generation method in one embodiment;

[0033] Figure 5 is a flow chart of a speech generation method in another embodiment;

[0034] Figure 6 is a structural block diagram of a speech generating device in one embodiment;

[0035] Figure 7 is a structural block diagram of a speech generating device in another embodiment;

[0036] Figure 8 is an internal structure diagram of a computer device in one embodiment;

[0037] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] The speech generation method provided in this application can be applied to Figure 1 In the application environment shown. In this application scenario, the computer device 102 and the database 104 communicate through a network. The database 104 pre-stores the voice features of the second user under various emotion types. The computer device 102 will obtain the image of the first user, and identify the state of the first user based on the image of the first user to obtain the state features of the first user. The computer device 102 can identify the emotion type of the first user based on the state features, and determine the response emotion type matching the emotion type. The computer device 102 can obtain the response sound features of the second user corresponding to the response emotion type from the database 104. The computer device 102 can obtain the response emotion features corresponding to the response emotion type; obtain the response text content matching the state features, and extract the content features of the response text content; fuse the content features, the response emotion features and the response sound features to obtain the fusion features. The computer device 102 can perform speech synthesis based on the fusion features to generate the response speech of the second user under the response emotion type and used to express the response text content.

[0040] It is understandable that the computer device 102 can be a terminal or a server, and there is no limitation on this. The server can be implemented as an independent server or a server cluster composed of multiple servers. The terminal can be an intelligent robot. The present application does not limit the specific form of the intelligent robot, which can be an anthropomorphic robot or a non-anthropomorphic terminal device, such as at least one of various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, as long as it can output voice to communicate with the user.

[0041] In one embodiment, Figure 2 As shown, a speech generation method is provided. In this embodiment, the method is applied to Figure 1 The computer device in the example is described, and the computer device can be a terminal or a server. It can be understood that the method can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0042] Step 202: Identify the state of the first user based on the image of the first user to obtain the state feature of the first user.

[0043] It can be understood that the image of the first user includes image content for presenting the first user, that is, the image of the first user is displayed in the image of the first user, so that the state of the first user can be identified therefrom.

[0044] The state of the first user refers to the state of the first user in the image, that is, the state of the first user when the first image is captured. It can be understood that the state of the first user identified from the image may include any visual state that can express emotions, such as facial expression state and body state. The state feature of the first user is a feature used to characterize the state of the first user.

[0045] Specifically, the computer device may perform feature extraction processing on the image of the first user, and identify the state of the first user based on the extracted image features to obtain the state features of the first user.

[0046] It should be noted that the state characteristics of the first user can be determined only based on the state of the first user identified from the image. In other embodiments, the state characteristics of the first user, in addition to being determined based on the state of the first user identified in the image, can also be determined in combination with other data that can express the emotions of the first user. For example, the voice of the first user can also express his emotions. Therefore, the state characteristics of the first user can also be determined in combination with the voice data of the first user in the image of the first user. Therefore, the state characteristics of the first user may include at least one of the visual state characteristics and sound characteristics in the image of the first user. There is no limitation on this. It can be understood that the visual state characteristics include at least one of the facial expression characteristics and body characteristics of the first user in the image.

[0047] Step 204: Identify the emotion type of the first user based on the state feature, and determine a coping emotion type that matches the emotion type.

[0048] The emotion type of the first user is used to characterize the emotion of the first user when the image of the first user is collected. The response emotion type is an emotion type used to respond to the emotion type of the first user. For example, if the first user is a child and the emotion type of the child is "crying", then the response emotion type may be a "peaceful" emotion type to respond to the child's crying emotion.

[0049] Specifically, the computer device may perform emotion recognition processing based on the obtained state characteristics of the first user to identify the emotion type of the first user.

[0050] In one embodiment, the computer device may determine a coping emotion type that matches the emotion type of the first user based on a pre-established coping relationship between emotion types.

[0051] In another embodiment, the computer device may also input the identified emotion type of the first user into a pre-trained emotion response model, and output a response emotion type that matches the emotion type of the first user. It can be understood that the emotion response model may be a machine learning model that is pre-trained based on sample emotion pairs and is able to predict the response emotion type. A sample emotion pair is two sample emotion types, and there is a response relationship between the two sample emotion types.

[0052] Step 206, obtaining a response sound feature of the second user corresponding to the response emotion type; obtaining a response emotion feature corresponding to the response emotion type.

[0053] Among them, the second user is a user who responds to the emotions of the first user. The second user can be a user who has a close relationship with the first user, such as a relative or friend of the first user. It can be understood that users who have a close relationship with the first user can often effectively respond to the emotions of the first user. The second user can also be a user of other identities who can effectively respond to the emotions of the first user. For example, the second user can be a psychological counselor of the first user or a person of other identities, and this is not limited.

[0054] In one embodiment, there may be multiple preset response users for responding to the emotions of the first user, and the computer device may select at least one of the multiple preset response users as the second user based on the emotion type of the first user. That is, the second user may also be one or more. For example, if the first user is a child, the corresponding preset response users may include father, mother, grandparents, brothers and sisters, etc. If the child cries, the mother can be selected as the second user to generate the mother's words to comfort the child.

[0055] The response sound feature refers to the sound feature of the second user in the response emotion type. It should be noted that the sound feature of the second user in each embodiment of the present application is a feature that can reflect the emotion type and can be characterized as the voice of the second user. Therefore, the response sound feature is a feature that can reflect the response emotion type and can be characterized as the voice of the second user. It can be understood that the second user has different sound features under different emotion types.

[0056] Specifically, the voice features of the second user under multiple emotion types are pre-stored, and the computer device can obtain the second user's response voice features corresponding to the response emotion type. The computer device can obtain the response emotion feature corresponding to the response emotion type. The response emotion feature is feature encoding data that can characterize the response emotion type.

[0057] It is understandable that the computer device can perform feature encoding on the coping emotion type to generate the coping emotion feature, or can search for the coping emotion feature corresponding to the coping emotion type according to the correspondence between the pre-stored emotion type and the emotion feature, and there is no limitation on this.

[0058] Step 208, obtaining the response text content that matches the state feature, and extracting the content feature of the response text content.

[0059] Among them, the response text content that matches the state feature refers to the text content used to respond to the state of the first user. That is, the state of the first user needs to be responded to, and in this case, certain text content is needed to respond. Now for example, when a child cries, the parents need to say something to respond to the child's crying state, for example, they will say "crying can't solve the problem", then the sentence "crying can't solve the problem" is the response text content.

[0060] Specifically, the computer device can directly obtain the response text content that matches the state feature, or generate the response text content after analyzing the state feature. The computer device can perform feature extraction on the response text content to obtain content features. It can be understood that the content features can reflect the semantics of the response text content.

[0061] In one embodiment, a plurality of response text contents are pre-stored in the text database. The computer device can obtain the response text contents matching the state features from the text database. Specifically, the computer device can obtain the response text contents matching the state features from the text database according to the response relationship between the pre-stored state features and the response text contents.

[0062] In another embodiment, the computer device may analyze the state features to generate response text content. Specifically, the computer device may input the state features into a pre-trained text generation model to output response text content matching the state features based on the text generation model. The text generation model is a machine learning model that is iteratively trained using multiple sets of sample state features and sample response texts as training data in advance, and is used to predict response text content matching the state features.

[0063] Step 210, fusing the content feature, the response emotion feature and the response sound feature to obtain a fusion feature.

[0064] Specifically, the computer device can combine the content feature, the response emotion feature and the response voice feature to obtain a fusion feature. It can be understood that the fusion feature can reflect the emotion of the response, the semantics of the text content to be expressed, and the voice feature of the second user who responds.

[0065] In one embodiment, the computer device may directly splice the fused content feature, the coping emotion feature, and the coping sound feature to obtain the fused feature. In another embodiment, the computer device may also replace some of the specified characters in the content feature with the coping emotion feature and the coping sound feature, thereby fusing and embedding the coping emotion feature and the coping sound feature into the content feature to form the fused feature.

[0066] Step 212: Perform speech synthesis based on the fusion features to generate a response speech of the second user in the response emotion type and used to express the response text content.

[0067] Specifically, the computer device can input the fusion feature into the speech synthesis network to perform speech synthesis processing to generate the response speech of the second user under the response emotion type. It can be understood that the response speech can reflect the response emotion type and express the response text content in the name of the second user. That is, the response speech is a simulated speech that simulates the second user expressing the response text content under the emotion of the response emotion type.

[0068] For example, if the response emotion type is "peaceful" and the response text content is "crying won't solve the problem", then the response voice expression is to simulate the father's voice and say the sentence "crying won't solve the problem" in a calm emotion.

[0069] The above-mentioned speech generation method identifies the state of the first user based on the image of the first user, and identifies the emotion type of the first user based on the identified state features. Then, determine the response emotion type that matches the emotion type, obtain the response sound features and response emotion features of the second user corresponding to the response emotion type, and determine the matching response text content according to the state features of the first user. The content features, response emotion features and response sound features of the text content are integrated, and speech synthesis is performed based on the integrated fusion features, so that a response speech can be generated that responds to the text content with the voice expression of the second user under the response emotion type. Compared with the traditional preset content with a fixed preset sound, it can adaptively generate a response speech for more accurately responding to the current state of the first user, which greatly improves flexibility, avoids limitations, and reduces the waste of computer resources to a certain extent.

[0070] In one embodiment, determining the coping emotion type that matches the emotion type in step 204 includes: searching the emotion library for the emotion type of the second user that is used to cope with the emotion type of the first user according to the coping relationship to obtain the coping emotion type.

[0071] Specifically, an emotion library is pre-set, and the response relationship between emotion types is pre-stored in the emotion library. Based on the response relationship, the computer device can search the emotion library for the emotion type of the second user that is used to respond to the emotion type of the first user to obtain the response emotion type.

[0072] For example, the emotion types a, b, c~n of the child are marked with the emotion types A, B, C~N of the relative. For example, the emotion type A "calm" of the relative is used to deal with the emotion type a "crying" of the child. In one embodiment, the response relationship between the emotion types stored in the emotion library may not be different for different second users. That is, the emotion library stores various emotion types used to deal with the emotion type of the first user.

[0073] For example, when the child's emotion type is "crying", the emotion type used to respond to "crying" can be "calm", and whether the second user is a father or a mother does not affect the response emotion type being "calm". If there are multiple preset response users for response, the preset response user that best matches the emotion type of the first user can be selected as the second user.

[0074] In another embodiment, if there are multiple preset response users, different emotion type response relationships can be stored in the emotion library for different preset response users. That is, different preset response users can respond to the same emotion type of the first user with different emotions. For example, in response to a child crying, the father's response emotion type can be "harsh", and the mother's response emotion type can be "peaceful". Therefore, the computer device can first determine the preset response user that best matches the emotion type of the first user as the second user, and then determine the response emotion type of the second user to respond to the emotion type of the first user based on the response relationship between the emotion types preset for the second user.

[0075] In the above embodiment, the response relationship between emotion types is pre-stored in the emotion library. Based on the emotion library, the emotion type suitable for responding to the current emotion of the first user can be quickly determined, which not only improves the efficiency, but also improves the accuracy of generating response speech to a certain extent, thereby reducing the waste of computer resources.

[0076] Figure 3 FIG. 1 is a simplified diagram of a speech generation method in one embodiment. Figure 3It can be seen that after capturing the image of the first user, the image is subjected to image recognition, and based on the identified state features of the first user, the corresponding text content is searched from the text database, and the emotional type of the first user is determined based on the state features of the first user, thereby searching the corresponding emotional type and the corresponding sound features of the second user under the corresponding emotional type from the emotional database. The content features of the corresponding text content, the corresponding emotional features of the corresponding emotional type, and the corresponding sound features of the second user are input into the voice conversion unit for fusion, and the voice conversion unit generates and outputs the second user's emotional response voice based on the fused fusion features.

[0077] It can be understood that the pre-set text database and emotion database can easily and accurately determine the response text, response emotion and the response sound characteristics of the second user, so that the characteristics of these three are integrated to synthesize the response voice of the second user, which greatly improves the response efficiency and response accuracy, thereby saving computer resources.

[0078] In one embodiment, the state feature of the first user includes the expression feature of the first user. Determining the response emotion type that matches the emotion type in step 204 includes: determining multiple candidate emotion types for responding to the emotion type of the first user; obtaining the expression feature of the second user under each candidate emotion type; matching the expression feature of the first user with the expression feature of the second user under each candidate emotion type; and determining the candidate emotion type to which the matched expression feature of the second user belongs as the response emotion type.

[0079] Specifically, there are multiple emotion types for coping with the emotion type of the first user, and the computer device can determine these multiple emotion types as candidate emotion types. It can be understood that the expressions of users under different emotions are usually different, so the computer device can obtain the expression features of the second user under each candidate emotion type, that is, obtain the expression features of the second user under each candidate emotion type. The computer device can match the expression features of the first user identified from the image of the first user with the expression features of the second user under each candidate emotion type to determine the expression features of the second user that match the expression features of the first user. The computer device can determine the candidate emotion type to which the matched expression features of the second user belong as the coping emotion type.

[0080] In the above embodiment, when determining the response emotion type, in addition to considering the emotion type of the first user, the facial features of the first user are also combined, and the facial features of the first user and the facial features of the candidate emotion type of the second user are advanced matched, thereby improving the accuracy of the determined response emotion type. Furthermore, the accuracy of the synthesized response speech is improved, and the waste of computer resources is reduced to a certain extent.

[0081] In one embodiment, obtaining the expression feature of the second user under each candidate emotion type includes: obtaining the expression feature of the second user corresponding to each candidate emotion type stored in the emotion library. In this embodiment, obtaining the response sound feature of the second user corresponding to the response emotion type includes: obtaining the response sound feature of the second user corresponding to the response emotion type stored in the emotion library.

[0082] Specifically, the emotion library pre-stores the emotion type of the second user, and the corresponding relationship between the emotion type of the second user and the emotion type of the first user, and stores the corresponding expression features and sound features of the second user for each emotion type of the second user. The computer device can determine the emotion type of the second user corresponding to the emotion type of the first user based on the corresponding relationship stored in the emotion library. In the case where there are multiple emotion types of the determined second user, the computer device can use the multiple determined emotion types as candidate emotion types. Then, the expression features of the corresponding second user stored for each candidate emotion type are obtained from the emotion library. The computer device can match the expression features of the first user with the expression features of the corresponding second user stored for each candidate emotion type obtained from the emotion library, and determine the candidate emotion type to which the matched expression features of the second user belong as the corresponding emotion type. The computer device can then obtain the sound features of the corresponding second user stored for the corresponding emotion type from the emotion library to obtain the corresponding sound features.

[0083] In the above embodiment, in the emotion library, the corresponding second user's expression features and voice features are pre-stored for the emotion type, so that the corresponding emotion type and the corresponding voice features can be determined conveniently and accurately, which greatly improves efficiency.

[0084] In one embodiment, before obtaining the response sound characteristics of the second user corresponding to the response emotion type, the method also includes the step of obtaining the sound characteristics of the second user under multiple emotion types, specifically including the following steps: obtaining sample speech of the second user under different emotion types; iteratively training a speech feature extraction model based on the sample speech of the second user under each emotion type, so that the speech feature extraction model iteratively learns the sound characteristic information of the second user under each emotion type from the sample speech, so as to iteratively update the sound feature coding information of the second user under the corresponding emotion type based on the sound characteristic information; obtaining the sound feature coding information of the second user under each emotion type after the training is completed, and obtaining the sound characteristics of the second user under each emotion type.

[0085] Specifically, the second user can be allowed to make corresponding sounds under different emotion types in advance, so as to obtain sample speech of the second user under multiple emotion types. The computer device has a speech feature extraction model to be trained, which contains the initialized sound feature coding information of the second user. It can be understood that the initialized sound feature coding information does not have the actual sound features of the second user. Therefore, the speech feature extraction model can be iteratively trained based on the sample speech of the second user under each emotion type, so that the speech feature extraction model can iteratively learn the sound characteristics of the second user under each emotion type from the sample speech, so as to iteratively update the sound feature coding information of the second user under the corresponding emotion type. In this way, after the iterative training is completed, the final sound feature coding information of the second user under each emotion type can be obtained. It can be understood that the sound feature coding information of the second user after the training has fully learned the sound characteristics of the second user when speaking, and thus belongs to the accurate sound features of the second user under each emotion type.

[0086] In one embodiment, for sample speech under each emotion type, when the computer device uses the sample speech to train the speech feature extraction model, the speech feature extraction model to be trained can identify the phoneme posterior probability features and fundamental frequency parameter features from the sample speech, which is equivalent to learning the sound characteristics, splicing the phoneme posterior probability features and fundamental frequency parameter features, and then, the spliced ​​features are sequentially processed by the conditional network and upsampling to generate the final sound feature encoding information of the second user under the emotion type.

[0087] Figure 4 FIG. 1 is a schematic diagram showing the principle of a speech generation method in an embodiment. Figure 4It can be seen that the sample speech of the second user under each emotion type is pre-stored in the database. It can be understood that the voice features of the second user are needed to generate the response speech, so the second user is equivalent to the target speaker. The sample speech of the second user under each emotion type can be feature extracted by the speech feature extraction model to obtain the spectrum parameter features and the fundamental frequency parameter features, and the phoneme posterior probability extractor in the speech feature extraction model generates the phoneme posterior probability features based on the spectrum parameter features, and then the phoneme posterior probability features and the fundamental frequency parameter features are spliced, and then the sound features of the second user under various emotion types are generated after upsampling through the conditional network. Further, after determining the response emotion type that matches the emotion type of the first user, the response sound features of the second user under the response emotion type can be obtained from the sound features extracted based on the sample speech. Then, the content features of the response text content, the response emotion features corresponding to the response emotion type, and the response sound features are spliced ​​to generate fusion features, and the fusion features are input into the speech synthesis network for speech synthesis to generate a speech waveform signal for simulating the second user under the response emotion type. It can be understood that by inputting the data output by the speech synthesis network into the sound decoder for decoding, the simulated response voice of the second user under the response emotion type can be played. During the model training process, other model parameters such as the conditional network can be fixed, and only the model parameters of the network used for feature extraction can be iteratively trained to improve its ability to learn sound characteristics.

[0088] In another embodiment, the corresponding embedded coding information can be initialized in advance for the timbre expression dimension. During the training process of the speech feature extraction model, the computer device can extract characteristic information of the timbre expression from the sample speech to iteratively update the embedded coding information, and extract the phoneme posterior probability feature and fundamental frequency parameter feature from the sample speech, and splice the phoneme posterior probability feature and fundamental frequency parameter feature, and splice the embedded coding information representing the timbre expression characteristics. Then, the spliced ​​features are sequentially processed by the conditional network and upsampling to generate the final sound feature coding information of the second user under the emotion type. Thus, the characteristics of the second user's voice are extracted from multiple aspects such as timbre expression, linguistic content, and rhythm, so that more accurate sound features can be obtained, and then in the subsequent speech synthesis processing stage, a response speech that is closer to the second user's original voice can be synthesized.

[0089] In one embodiment, the state feature includes the expression feature and the voice feature of the first user. In this embodiment, step 202 recognizes the state of the first user based on the image of the first user, and obtaining the state feature of the first user includes: obtaining the captured image of the first user; obtaining the voice data of the first user when the image is captured; recognizing the expression state of the first user from the image to obtain the expression feature of the first user; recognizing the voice state of the first user from the voice data to obtain the voice feature of the first user.

[0090] It can be understood that the collected image and sound data are corresponding data occurring at the same time and in the same space, and can accurately reflect the state of the first user at that time. The first user's expression and sound can reflect the first user's emotions to a certain extent, and therefore, the first user's emotion type can be more accurately identified based on the first user's expression features and sound features.

[0091] In one embodiment, the computer device may have image acquisition capability, that is, the computer device may acquire an image of the first user and acquire voice data of the first user when acquiring the image.

[0092] In another embodiment, the computer device may have both image acquisition capability and voice acquisition capability. For example, the computer device may be a companion robot with both image acquisition and voice acquisition functions. That is, the computer device may not only acquire the image of the first user, but also acquire the voice data of the first user when acquiring the image.

[0093] In other embodiments, the computer device may not have the image acquisition and voice acquisition capabilities, but executes the method of the present application based on the acquired image and sound data to synthesize the response voice. This is not limited to this.

[0094] It can be understood that jointly determining the state of the first user based on the first user's expression and voice improves the accuracy of determining the state characteristics, thereby also improving the accuracy of recognizing the first user's current (i.e., when the image of the first user is captured) emotion type, and ultimately improving the accuracy of the synthesized response speech.

[0095] like Figure 5 As shown, in one embodiment, another speech generation method is provided, which specifically includes the following steps:

[0096] Step 502: extracting the voice features and expression features of the second user under different emotion types from the sample voices and sample expressions of the second user under different emotion types respectively.

[0097] Step 504: In the emotion library, the emotion type of the second user, the expression features and the voice features of the second user are correspondingly stored, as well as the corresponding relationship between the emotion type of the first user and the emotion type of the second user is stored.

[0098] It can be understood that for each emotion type of the second user, the corresponding expression features and voice features of the second user are stored, and the response relationships between the emotion types are stored.

[0099] Step 506, obtaining the captured image of the first user and obtaining the voice data of the first user when the image is captured; identifying the expression state of the first user from the image to obtain the expression characteristics of the first user; identifying the voice state of the first user from the voice data to obtain the voice characteristics of the first user.

[0100] Step 508, identifying the emotion type of the first user based on the facial expression features and voice features of the first user; and determining multiple candidate emotion types for responding to the emotion type of the first user from the emotion library according to the response relationship between the emotion type of the first user and the emotion type of the second user stored in the emotion library.

[0101] Step 510 , acquiring the expression features of the second user under each candidate emotion type from the emotion library according to the correspondence between the emotion type of the second user, the expression features of the second user, and the sound features stored in the emotion library.

[0102] Step 512, matching the first user's facial expression features with the second user's facial expression features under each candidate emotion type respectively; and determining the candidate emotion type to which the matched second user's facial expression features belong as the response emotion type.

[0103] Step 514, according to the correspondence between the emotion type of the second user, the expression features of the second user and the sound features stored in the emotion library, obtain the sound features of the second user corresponding to the response emotion type from the emotion library to obtain the response sound features.

[0104] Step 516, feature encoding is performed on the response emotion type to obtain the response emotion feature; the response text content matching the state feature is obtained from the text database, and feature encoding is performed on the response text content to obtain the content feature.

[0105] Step 518, concatenate and fuse the content features, the response emotion features, and the response sound features to obtain a fusion feature.

[0106] Step 520: Perform speech synthesis based on the fusion features to generate a response speech of the second user under the response emotion type and used to express the response text content.

[0107] It should be understood that, although each step in the flow chart of each embodiment of the present application is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have strict order restrictions, and these steps can be performed in other orders. Moreover, at least a portion of the steps of the flow chart of each embodiment of the present application may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0108] like Figure 6 As shown, in one embodiment, a speech generation device is provided, the device comprising: a state recognition module 602, a coping emotion determination module 604, a feature acquisition module 606, a feature fusion module 608 and a speech synthesis module 610; wherein:

[0109] The state recognition module 602 is used to recognize the state of the first user based on the image of the first user to obtain the state characteristics of the first user; the image includes image content for presenting the first user.

[0110] The coping emotion determination module 604 is used to identify the emotion type of the first user based on the state characteristics, and determine a coping emotion type that matches the emotion type.

[0111] The feature acquisition module 606 is used to obtain the response sound features of the second user corresponding to the response emotion type; obtain the response emotion features corresponding to the response emotion type; obtain the response text content matching the state features, and extract the content features of the response text content.

[0112] The feature fusion module 608 is used to fuse content features, response emotion features and response sound features to obtain fused features.

[0113] The speech synthesis module 610 is used to perform speech synthesis based on the fusion features to generate a response speech of the second user under the response emotion type and used to express the response text content.

[0114] In one embodiment, the coping emotion type is determined from an emotion library; the emotion library pre-stores the coping relationship between emotion types. The coping emotion determination module 604 is also used to search the emotion library for the emotion type of the second user for coping with the emotion type of the first user according to the coping relationship, and obtain the coping emotion type.

[0115] In one embodiment, the state feature includes an expression feature of the first user. The response emotion determination module 604 is further used to determine multiple candidate emotion types for responding to the emotion type of the first user; obtain the expression feature of the second user under each candidate emotion type; match the expression feature of the first user with the expression feature of the second user under each candidate emotion type; and determine the candidate emotion type to which the matched expression feature of the second user belongs as the response emotion type.

[0116] In one embodiment, the candidate emotion type is determined from the emotion types of the second user pre-stored in the emotion library; the emotion library also stores the expression features and voice features of the second user corresponding to each emotion type of the second user. The feature acquisition module 606 is also used to acquire the expression features of the second user corresponding to each candidate emotion type stored in the emotion library; and acquire the voice features of the second user corresponding to the response emotion type stored in the emotion library to obtain the response voice features.

[0117] like Figure 7 As shown, in one embodiment, the device also includes:

[0118] The feature extraction module 601 is used to obtain sample speech of the second user under different emotion types; iteratively train the speech feature extraction model based on the sample speech of the second user under each emotion type, so that the speech feature extraction model iteratively learns the sound characteristic information of the second user under each emotion type from the sample speech, and iteratively updates the sound feature coding information of the second user under the corresponding emotion type based on the sound characteristic information; obtain the sound feature coding information of the second user under each emotion type after the training is completed, and obtain the sound characteristics of the second user under each emotion type.

[0119] In one embodiment, the state feature includes the expression feature and voice feature of the first user. The state recognition module 602 is further used to obtain the captured image of the first user; obtain the voice data of the first user when the image is captured; recognize the expression state of the first user from the image to obtain the expression feature of the first user; recognize the voice state of the first user from the voice data to obtain the voice feature of the first user.

[0120] For the specific definition of the speech generation device, please refer to the definition of the speech generation method above, which will not be repeated here. Each module in the above-mentioned speech generation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0121] In one embodiment, a computer device is provided, which may be a server. It is understood that the server may be, for example, a backend server for a home care robot to perform data analysis. Its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the voice features of the second user of each preset emotion type. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a speech generation method is implemented.

[0122] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9 As shown. The terminal may be, for example, a home care robot. The computer device comprises a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for generating speech is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0123] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0124] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0125] In some embodiments, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0126] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0127] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A speech generation method, characterized in that: The method comprises: recognizing the state of the first user based on the image of the first user, and obtaining the state characteristics of the first user; the image includes image content for presenting the first user; Identify the emotion type of the first user based on the state feature, and determine multiple candidate emotion types for coping with the emotion type of the first user; obtain the expression features of the second user under each of the candidate emotion types; match the expression features of the first user with the expression features of the second user under each candidate emotion type; determine the candidate emotion type to which the matched expression features of the second user belong as the coping emotion type; Acquiring a response sound feature of a second user corresponding to the response emotion type; Acquire a coping emotion feature corresponding to the coping emotion type; Acquire the response text content that matches the state feature, and extract the content feature of the response text content; fusing the content feature, the response emotion feature, and the response sound feature to obtain a fusion feature; Speech synthesis is performed based on the fusion features to generate a response speech of the second user under the response emotion type and used to express the response text content.

2. The method according to claim 1, characterized in that The coping emotion type is determined from the emotion library; The emotion library pre-stores the response relationship between emotion types; the method further includes: According to the response relationship, the emotion type of the second user used to respond to the emotion type of the first user is searched from the emotion library to obtain the response emotion type.

3. The method according to claim 1, characterized in that The candidate emotion type is determined from the emotion types of the second user pre-stored in the emotion library; the emotion library also stores the corresponding expression features and voice features of the second user for each emotion type of the second user; The acquiring of the facial expression feature of the second user under each of the candidate emotion types comprises: Acquire, from the emotion library, the expression feature of the second user corresponding to each candidate emotion type; The obtaining of the second user's response sound feature corresponding to the response emotion type includes: From the emotion library, the voice feature of the second user corresponding to the coping emotion type is obtained to obtain the coping voice feature.

4. The method according to claim 1, characterized in that: Before acquiring the response sound feature of the second user corresponding to the response emotion type, the method further includes: Obtaining sample speech of the second user under different emotion types; Iteratively training a speech feature extraction model based on the sample speech of the second user under each emotion type, so that the speech feature extraction model iteratively learns the voice characteristics of the second user under each emotion type from the sample speech, so as to iteratively update the voice feature encoding information of the second user under the corresponding emotion type; The voice feature encoding information of the second user under each emotion type after the training is completed is obtained to obtain the voice feature of the second user under each emotion type.

5. The method according to any one of claims 1 to 4, characterized in that The state characteristics include expression characteristics and voice characteristics of the first user; The identifying the state of the first user based on the image of the first user to obtain the state feature of the first user includes: Acquire a captured image of the first user; Acquiring voice data of the first user when capturing the image; Recognizing the expression state of the first user from the image to obtain the expression features of the first user; The voice state of the first user is identified from the voice data to obtain the voice feature of the first user.

6. The method according to claim 1, characterized in that Before acquiring the response sound feature of the second user corresponding to the response emotion type, the method further includes: By using a speech feature extraction model, feature extraction is performed on the sample speech of the second user under each emotion type to obtain spectrum parameter features and fundamental frequency parameter features; Generate phoneme posterior probability features based on spectral parameter features through a phoneme posterior probability extractor in the speech feature extraction model; The concatenation result of the phoneme posterior probability feature and the fundamental frequency parameter feature is upsampled through a conditional network to generate voice features of the second user under various emotion types.

7. A speech generating device, characterized in that: The device comprises: A state recognition module, configured to recognize the state of a first user based on an image of the first user, and obtain state characteristics of the first user; the image includes image content for presenting the first user; A response emotion determination module is used to identify the emotion type of the first user based on the state characteristics, and determine a plurality of candidate emotion types for responding to the emotion type of the first user; obtain the expression characteristics of the second user under each of the candidate emotion types; match the expression characteristics of the first user with the expression characteristics of the second user under each candidate emotion type; and determine the candidate emotion type to which the matched expression characteristics of the second user belong as the response emotion type; A feature acquisition module, used to acquire a response sound feature of the second user corresponding to the response emotion type; acquire a response emotion feature corresponding to the response emotion type; acquire a response text content matching the state feature, and extract a content feature of the response text content; A feature fusion module, used to fuse the content feature, the response emotion feature and the response sound feature to obtain a fusion feature; A speech synthesis module is used to perform speech synthesis based on the fusion feature to generate a response speech of the second user under the response emotion type and used to express the response text content.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images