Voice conversion method, device, computer equipment, storage medium and program product

Abstract

This application relates to a speech conversion method, apparatus, computer device, storage medium, and program product. The method includes: obtaining the speaker's body language and target speech information, where the target speech information represents speech information emitted through a specific vocalization state; determining the speaker's emotional state at the time the target speech information was emitted based on the body language; and performing speech conversion processing on the target speech information using an emotional speech acoustic model corresponding to the emotional state to obtain emotional speech information corresponding to the target speech information, where the emotional speech information represents speech conveying the emotional state. This method enables the recipient to correctly understand the meaning of the speaker's whispered expression.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech conversion method, apparatus, computer equipment, storage medium, and program product. Background Art

[0002] With the continuous development of communication technology, in environments where noise is prohibited, such as libraries, hospitals, or conference rooms, the speaker usually communicates with the receiver through whispering. Since the speaker's vocal cords do not vibrate when whispering, the receiver cannot hear the whispered content clearly, causing great trouble to both the speaker and the receiver.

[0003] In the related art, whispers are input into a whisper processing unit, which processes the whispers to obtain text information corresponding to the whispers. The text information is then output as a robot voice at a normal volume so that the recipient can clearly hear the whispers spoken by the sender.

[0004] However, the normal volume robot voice output by the related technology ignores the emotional expression of the speaker, which may cause the recipient to be unable to understand or misunderstand the meaning of the speaker's whispered expression. Summary of the Invention

[0005] Based on this, it is necessary to provide a speech conversion method, device, computer equipment, storage medium and program product that can enable the recipient to correctly understand the meaning of the whispered expression of the sender to address the above technical problems.

[0006] In a first aspect, the present application provides a method for voice conversion, the method comprising:

[0007] Acquiring the body language and target voice information of the speaker; the target voice information refers to the voice information sent through a specific voice state;

[0008] Determine the sender's emotional state when delivering the target voice message based on body language;

[0009] The target speech information is subjected to speech conversion processing by using the emotional speech acoustic model corresponding to the emotional state to obtain the emotional speech information corresponding to the target speech information; the emotional speech information represents the speech carrying the emotional state.

[0010] In one embodiment, body language includes body movements and facial expressions; determining the emotional state of the sender when sending the target voice message based on the body language includes:

[0011] Obtain the facial image and body key point position information of the speaker;

[0012] The facial expression of the speaker is extracted from the facial image, and the body movements of the speaker are determined through the position information of the key points of the body.

[0013] Determine the emotional state of the speaker based on facial expressions and body movements.

[0014] In one embodiment, the method further comprises:

[0015] Get the emotion label corresponding to the emotional state;

[0016] The emotion label is matched with the label in each emotional speech acoustic model, and the emotional speech acoustic model that is successfully matched is determined as the emotional speech acoustic model.

[0017] In one embodiment, performing voice conversion processing on target voice information to obtain emotional voice information corresponding to the target voice information includes:

[0018] Extracting text information from the target voice information to obtain text information of the target voice information;

[0019] The text information is input into the emotional speech acoustic model to obtain the emotional speech information corresponding to the target speech information.

[0020] In one embodiment, extracting text information from target voice information to obtain text information of the target voice information includes:

[0021] Extracting keyword text from target voice information;

[0022] Perform word expansion processing on the keyword text to generate text information of the target voice information.

[0023] In one embodiment, the keyword text is expanded to generate text information of the target voice information, including:

[0024] Perform search analysis on keyword texts to obtain multiple candidate search text information;

[0025] According to the retrieval text information with the highest similarity to the keyword text among the candidate retrieval text information, the keyword text is subjected to word fusion processing to obtain the text information of the target voice information.

[0026] In one embodiment, the process of constructing the emotional speech acoustic model includes:

[0027] Acquire multiple sample text information and sample emotional speech information with different emotional states corresponding to each sample text information;

[0028] Performing text analysis on each sample text information to obtain multiple labeled sample text information, performing parameter extraction on each sample emotional speech information to obtain acoustic parameters of each sample emotional speech information;

[0029] According to the acoustic parameters of each labeled sample text information and each sample emotional speech information, the initial emotional speech acoustic model is trained until a preset convergence condition is met, and the convergence of the initial emotional speech acoustic model is determined to obtain the emotional speech acoustic model.

[0030] In one embodiment, the convergence condition is that the similarity between the emotional speech information output by the initial emotional speech acoustic model and the corresponding sample emotional speech information is greater than a preset similarity threshold.

[0031] In a second aspect, the present application further provides a speech conversion device, the device comprising:

[0032] The first acquisition module is used to acquire the body language and target voice information of the speaker; the target voice information refers to the voice information sent through a specific voice state;

[0033] The first determination module is used to determine the emotional state of the sender when sending the target voice message based on body language;

[0034] The first processing module is used to perform speech conversion processing on the target speech information through the emotional speech acoustic model corresponding to the emotional state to obtain emotional speech information corresponding to the target speech information; the emotional speech information represents speech carrying an emotional state.

[0035] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the contents of all method embodiments in the first aspect are implemented.

[0036] In a fourth aspect, the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the contents of all method embodiments in the first aspect are implemented.

[0037] In a fifth aspect, the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the contents of all method embodiments in the first aspect are implemented.

[0038] The above-mentioned speech conversion method, apparatus, computer device, storage medium, and program product obtain the speaker's body language and target speech information, then determine the speaker's emotional state at the time the target speech information was sent based on the body language, then use the emotional speech acoustic model corresponding to the emotional state, and finally perform speech conversion processing on the target speech information to obtain the emotional speech information corresponding to the target speech information. In this method, the target speech information is speech information sent using a specific vocal state. If the target speech information is not processed, the recipient cannot understand the content of the target speech information. The speaker's body language can accurately determine the speaker's emotional state at the time the target speech information was sent, and the emotional speech acoustic model corresponding to the speaker's emotional state is accurately selected from multiple emotional speech acoustic models to perform speech conversion processing on the target speech information. Compared to traditional technologies that convert the target speech into speech produced by a robot, this method can ensure that the speech heard by the recipient is normal speech with emotion, enabling the recipient to correctly understand the meaning of the speaker's whispered expression, thereby improving the efficiency of communication between the two parties. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A diagram showing an application environment of a speech conversion method according to an embodiment;

[0040] Figure 2 1 is a flow chart of a method for voice conversion according to an embodiment;

[0041] Figure 3 1 is a flow chart of a method for voice conversion according to an embodiment;

[0042] Figure 4 1 is a flow chart of a method for voice conversion according to an embodiment;

[0043] Figure 5 1 is a flow chart of a method for voice conversion according to an embodiment;

[0044] Figure 6 1 is a flow chart of a method for voice conversion according to an embodiment;

[0045] Figure 7 1 is a flow chart of a method for voice conversion according to an embodiment;

[0046] Figure 8 1 is a flow chart of a method for voice conversion according to an embodiment;

[0047] Figure 9 1 is a flow chart of a method for voice conversion according to an embodiment;

[0048] Figure 10 1 is a flow chart of a method for voice conversion according to an embodiment;

[0049] Figure 11 FIG. 4 is a structural block diagram of a speech conversion device in one embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] The speech conversion method provided in the embodiment of the present application can be applied to Figure 1 The application environment shown in FIG. This application environment includes computer devices, which may include, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices.

[0052] In one embodiment, Figure 2 As shown, a speech conversion method is provided, which is applied to Figure 1 The computer device in the example is used to illustrate the process, including the following steps:

[0053] S201, obtaining the body language and target voice information of the speaker; the target voice information refers to the voice information sent through a specific voice state.

[0054] Body language refers to the use of body language to communicate with others, allowing them to understand our true thoughts. This can include eye contact and facial expressions, body movements and touch, posture and appearance, and the spatial distance between bodies. The aforementioned specific vocal states refer to the state of the larynx during vocalization, including the opening and closing of the vocal cords, tension or relaxation, vibration, and airflow. Specific vocal states can include whispers, normal voiced voices, falsetto, crackling sounds, and soft sounds caused by vocal cord vibration.

[0055] Specifically, the computer device can use its built-in video camera to record the speaker's body language and facial expressions over a preset time period, and determine the speaker's body language based on the body language and facial expressions. For example, body language movements can include waving hands, pushing hands outward, rubbing hands, tugging at the collar, shrugging shoulders, and supporting the chin with one hand. Facial expressions can include downturned corners of the mouth, tight lips, drooping eyebrows, and raised eyebrows. Simultaneously, the computer device can use its built-in audio capture device to capture the speaker's voice information over a preset time period, and determine this voice information as the target voice information. For example, this voice information can be whispered voice information or softly uttered by the speaker's vocal cords vibrating.

[0056] S202: Determine the emotional state of the sender when sending the target voice message based on the body language.

[0057] Among them, emotional states can include excitement, gratitude, regret, grief, anger, joy, sadness, confusion and fear.

[0058] Optionally, the computer device may use the emotional state corresponding to historical body language similar to the body language as the emotional state of the sender at the time the target voice message was sent. Alternatively, there may be a mapping relationship between body language and emotional state, and the computer device may determine the emotional state of the sender at the time the target voice message was sent based on the body language and the mapping relationship. This embodiment does not limit the method for determining the emotional state of the sender based on body language.

[0059] S203, performing speech conversion processing on the target speech information using the emotional speech acoustic model corresponding to the emotional state to obtain emotional speech information corresponding to the target speech information; the emotional speech information represents speech carrying the emotional state.

[0060] Each emotional state corresponds to an emotional speech acoustic model. When there are N emotional states in total, there are N emotional speech acoustic models. For example, the joyful emotional state corresponds to the joyful emotional speech acoustic model, and the frightened emotional state corresponds to the frightened emotional speech acoustic model.

[0061] Specifically, the computer device can input the target speech information into the emotional speech acoustic model, divide the target speech information into non-overlapping frames, perform pre-amplification filtering on the target speech information, perform linear prediction analysis on the filtered target speech information, adjust the gain according to the unvoiced and voiced sound types of the frame, and output the emotional speech information with emotion corresponding to the target speech information.

[0062] In the above-mentioned speech conversion method, the method obtains the speaker's body language and target speech information, then determines the speaker's emotional state at the time the target speech information is sent based on the body language, then uses the emotional speech acoustic model corresponding to the emotional state, and finally performs speech conversion processing on the target speech information to obtain the emotional speech information corresponding to the target speech information. In this method, the target speech information is speech information sent through a specific vocal state. If the target speech information is not processed, the recipient cannot understand the content of the target speech information. This method can accurately determine the speaker's emotional state at the time the target speech information is sent based on the speaker's body language, accurately select the emotional speech acoustic model corresponding to the speaker's emotional state from multiple emotional speech acoustic models, and perform speech conversion processing on the target speech information. Compared to traditional technologies that convert the target speech into speech produced by a robot, this method can ensure that the speech heard by the recipient is normal speech with emotion, and can enable the recipient to correctly understand the meaning of the speaker's whispered expression, thereby improving the efficiency of communication between the two parties.

[0063] Figure 3 This is a flow chart of the voice conversion method provided in the embodiment of the present application. The embodiment of the present application involves body language including body movements and facial expressions; it is an optional implementation method to determine the emotional state of the sender when sending the target voice message based on body language. Figure 2 Based on the embodiment shown, Figure 3 As shown, the above S201 may include the following steps:

[0064] S301, obtaining the facial image and body key point position information of the speaker.

[0065] Specifically, the computer device can detect the facial image area of ​​the speaker using an edge detection operator. The edge detection operator can be a Sobel operator, a Prewitt operator, a Canny operator, or the like. Since the speaker's facial area is the area with the most distinguishing points, the speaker's facial key points can be detected using a facial key point detection algorithm, and the speaker's facial contour area can be determined based on the facial key points. The facial key point detection algorithm can include an algorithm based on a point distribution model (PDM), an active appearance model (AAM), or a cascaded pose regression (CPR). For example, key points of facial information can include information such as interocular distance, eye width, and mouth width. Furthermore, the computer device can identify the positional information of the speaker's body key points using a feature classifier built into the Open Source Computer Vision Library (OpenCV).

[0066] S302, extracting the facial expression of the speaker through the facial image, and determining the body movements of the speaker through the position information of the key points of the body.

[0067] Specifically, facial expressions and body movements vary under different emotional states. A computer device can extract spatial feature information of multiple preset facial landmarks and positional feature information of key body points from a continuous image sequence of the speaker, and obtain change characteristics of specific facial regions of interest and positional features of key body points. For example, the speaker's facial expression could include downturned corners of the mouth, tight lips, drooping or raised eyebrows, and body movements could include waving hands, pushing hands outward, rubbing hands, tugging at the collar, shrugging shoulders, or supporting the chin with one hand.

[0068] S303, determining the emotional state of the sender based on facial expressions and body movements.

[0069] Specifically, the computer device can build a support vector machine (SVM) classifier based on the spatial feature information of multiple preset facial feature points, and determine the emotional state of the speaker based on the classification results of facial expressions. For example, with respect to facial expressions, when the speaker's facial expression is a downward-facing corner of the mouth, the body language indicates pain, sadness, or helplessness; when the speaker's facial expression is a tight lip, the body language indicates anger, confrontation, or determination; when the speaker's facial expression is a drooping eyebrow, the body language indicates helplessness, regret, or lack of interest; and when the speaker's facial expression is a raised eyebrow, the body language indicates annoyance, anger, or hatred. Furthermore, the computer device can build an SVM classifier based on the positional feature information of multiple key body points, and determine the emotional state of the speaker based on the classification results of body movements. For example, when it comes to body movements, when the person speaking waves his hands, the body language he expresses is to stop or deny; when the person speaking pushes his hands outward, the body language he expresses is to refuse; when the person speaking rubs his hands or pulls his collar, the body language he expresses is nervousness; when the person speaking shrugs his shoulders, the body language he expresses is indifference or helplessness; when the person speaking supports his chin with one hand, the body language he expresses is doubt.

[0070] Furthermore, it can be understood that when the classification result of facial expression is consistent with the classification result of body movement, the obtained emotional state will be used as the emotional state of the speaker; when the classification result of facial expression is inconsistent with the classification result of body movement, the classification result of facial expression shall prevail, and the emotional state obtained according to the classification result of facial expression will be used as the emotional state of the speaker.

[0071] The aforementioned speech conversion method obtains the speaker's facial image and body key point location information, then extracts the speaker's facial expression from the facial image, determines the speaker's body movements using the body key point location information, and finally determines the speaker's emotional state based on the facial expression and body movements. This method determines the speaker's emotional state based on the facial image and body key point location information, then fuses these two emotional states to evaluate them from two perspectives. Compared to methods that rely solely on facial expression or body key points, the resulting emotional state is more consistent with the speaker's actual emotional state, improving the accuracy of the emotional state.

[0072] Figure 4 The flow chart of the speech conversion method provided in the embodiment of the present application. The embodiment of the present application relates to an optional implementation method of determining the emotional speech acoustic model according to the emotional state. Figure 2 Based on the embodiment shown, Figure 4As shown, the above method may further include the following steps:

[0073] S401, obtaining an emotion label corresponding to an emotional state.

[0074] Specifically, different emotional states correspond to different emotion labels, with a one-to-one mapping relationship between emotional states and emotion labels. For example, when the emotional state is excitement, the corresponding emotion label is A; when the emotional state is sadness, the corresponding emotion label is B; when the emotional state is joy, the corresponding emotion label is C. After the computer device determines the sender's emotional state based on body language, it can determine the corresponding emotion label based on the mapping relationship between emotional states and emotion labels.

[0075] S402, matching the emotion label with the label in each emotional speech acoustic model, and determining the emotional speech acoustic model that is successfully matched as the emotional speech acoustic model.

[0076] Specifically, different emotional speech acoustic models correspond to different labels, and there is a one-to-one mapping between the emotional speech acoustic models and the labels. The labels are consistent with the labels corresponding to the emotional state. The computer device can match the emotional labels with the labels in each emotional speech acoustic model and use the successfully matched emotional speech acoustic model as the emotional speech acoustic model corresponding to the emotional state.

[0077] In the aforementioned speech conversion method, the method obtains an emotion tag corresponding to an emotional state, matches the emotion tag with the tag in each emotional speech acoustic model, and determines the successfully matched emotional speech acoustic model as the emotional speech acoustic model. This method accurately obtains the emotional speech acoustic model corresponding to the emotional state through the matching process of two emotion tags.

[0078] Figure 5 The flow chart of the speech conversion method provided in the embodiment of the present application. The embodiment of the present application relates to an optional implementation method of performing speech conversion processing on target speech information to obtain emotional speech information corresponding to the target speech information. Figure 2 Based on the embodiment shown, Figure 5 As shown, the above S203 may include the following steps:

[0079] S501: Extract text information from target voice information to obtain text information of the target voice information.

[0080] Optionally, the computer device may extract the text information from the target voice information using a corresponding text extraction algorithm to obtain text information corresponding to the target voice information. The text extraction algorithm may be a dynamic time warping (DTW) algorithm. Optionally, the computer device may input the target voice information into a preset neural network model and output the text information corresponding to the target voice information through calculations of the neural network model. This embodiment does not limit the method for obtaining text information from the target voice information.

[0081] Furthermore, it is understandable that before extracting the text information from the target voice information, the silent voice information at the beginning and end of the target voice information can be cut off, so as to compress the size of the voice information file.

[0082] Optional, Figure 6 The present invention provides a flow chart of the speech conversion method. The present invention relates to an optional implementation method of extracting text information from target speech information to obtain text information of the target speech information. Figure 5 Based on the embodiment shown, Figure 6 As shown, the above S501 may include the following steps:

[0083] S601: extracting keyword text from target voice information.

[0084] Specifically, because the target voice information is emitted in a specific voice state, the text information corresponding to the target voice information may be misrecognized during the conversion of the target voice information into text information. Therefore, it is necessary to extract keyword text information from the target voice information. The computer device can extract keyword text from the target text corresponding to the target voice information. For example, if the target voice information is "Determine the emotional state of the speaker based on facial expressions," the keyword text is "facial expression," "speaker," and "emotional state."

[0085] S602: Perform word expansion processing on the keyword text to generate text information of the target voice information.

[0086] Optionally, the computer device may input the keyword text information into a preset neural network model, and output text information containing the keyword through calculations by the neural network model. Alternatively, the computer device may search for text with the highest similarity to the keyword text information and expand the keyword text based on the text to obtain text information corresponding to the target voice information. This embodiment does not limit the method of word expansion processing.

[0087] In the aforementioned speech conversion method, the method extracts keyword text from the target speech information and performs word expansion on the keyword text to generate text information of the target speech information. Because the target speech information in this method is emitted in a specific voice state, some words may not be clear during the process of converting the target speech information into text information. Therefore, it is necessary to first extract the keyword text from the target speech information and then perform word expansion on the keyword text to make the resulting text information of the target speech information more closely resemble the content of the target speech information.

[0088] S502: Input the text information into the emotional speech acoustic model to obtain emotional speech information corresponding to the target speech information.

[0089] Specifically, the computer device takes text information as input information, inputs the text information into the emotional speech acoustic model, and after calculation by the emotional speech acoustic model, outputs the emotional speech information corresponding to the text information, and uses the speech information as the emotional speech information corresponding to the target speech information.

[0090] In the aforementioned speech conversion method, the method extracts text from the target speech information to obtain the text of the target speech information, and then inputs the text into an emotional speech acoustic model to obtain the emotional speech information corresponding to the target speech information. This method first extracts the text from the target speech information, presents the content of the target speech information in text form, and then uses the emotional speech acoustic model to output this text information as emotionally charged speech information. This allows the recipient to clearly hear the target speech information while also more easily understanding the intended meaning of the sender's target speech information.

[0091] Optional, Figure 7 The present invention provides a flow chart of the speech conversion method. The present invention relates to an optional implementation method of expanding the keyword text to generate text information of the target speech information. Figure 5 Based on the embodiment shown, Figure 7 As shown, the above S502 may include the following steps:

[0092] S701: Perform search analysis on the keyword text to obtain multiple candidate search text information.

[0093] Specifically, the computer device can input all keyword texts into the search box for search. When no candidate search text information is retrieved or the retrieved candidate text information is small, the number of keyword texts is reduced and the search is performed again until multiple candidate search text information about the keyword text is obtained.

[0094] S702 : performing word fusion processing on the keyword text according to the search text information with the highest similarity to the keyword text among the candidate search text information to obtain text information of the target voice information.

[0095] Optionally, after obtaining multiple candidate retrieval text information, the computer device calculates the similarity between the multiple candidate retrieval text information and the keyword text information, or selects the first one of the multiple candidate retrieval text information as the retrieval text information with the highest similarity to the keyword text, processes the keyword text information according to the candidate retrieval text information with the highest similarity, fills in the keyword text information, obtains complete text information corresponding to the keyword text information, and uses the text information as the text information of the target voice information.

[0096] In the above-mentioned speech conversion method, a keyword text is searched and analyzed to obtain multiple candidate search text information. Based on the search text information with the highest similarity to the keyword text among the candidate search text information, the keyword text is subjected to word fusion processing to obtain the text information of the target speech information. This method fuses the keyword text information with the text information with the highest keyword text similarity, making the expanded text information more closely resemble the content corresponding to the target speech information.

[0097] Figure 8 The present invention provides a flow chart of the speech conversion method. The present invention relates to an optional implementation of the process of constructing an emotional speech acoustic model. Figure 2 Based on the embodiment shown, Figure 8 As shown, the process includes the following steps:

[0098] S801: Acquire multiple sample text information and sample emotional speech information with different emotional states corresponding to each sample text information.

[0099] Specifically, different sample text messages correspond to different emotional states, and sample emotional voice messages carrying different emotional states are emitted based on the different emotions. For example, if the theme of a sample text message is about enjoying life, the corresponding emotional state is happiness; if the theme of a sample text message is about condemning a criminal, the corresponding emotional state is anger. A computer device can obtain multiple sample text messages from a webpage, and can also obtain sample emotional voice messages corresponding to the emotional states of the sample text messages.

[0100] S802 , performing text analysis processing on each sample text information to obtain a plurality of labeled sample text information, performing parameter extraction on each sample emotional speech information to obtain acoustic parameters of each sample emotional speech information.

[0101] Among them, acoustic parameters refer to the timbre of the sound and the frequency of the sound.

[0102] Specifically, the computer device can annotate multiple sample text information, classify sample text information belonging to the same emotional state into one category, and use the same annotation to annotate the corresponding sample text information. The computer device can annotate each sample text information through corresponding annotation software, or manually annotate each sample text information. Optionally, the computer device can extract the timbre and frequency from each sample emotional speech information through a deep learning model, or the computer device can extract the timbre and frequency from each sample emotional speech information through a principal component analysis algorithm (PCA), and use the extracted timbre and frequency as the acoustic parameters of each sample emotional speech information.

[0103] S803: Train the initial emotional speech acoustic model according to the acoustic parameters of each labeled sample text information and each sample emotional speech information until a preset convergence condition is met, determine that the initial emotional speech acoustic model converges, and obtain the emotional speech acoustic model.

[0104] Specifically, the acoustic parameters of each labeled sample text information and each sample emotional speech information are input into the initial emotional speech acoustic model, and the parameters of each layer of the initial emotional speech acoustic model are optimized and updated using a stochastic gradient descent algorithm. When the loss function converges, the emotional speech acoustic model is obtained, and the emotional speech acoustic model is used as the emotional speech acoustic model for the emotional state. The convergence condition is that the similarity between the emotional speech information output by the initial emotional speech acoustic model and the corresponding sample emotional speech information is greater than a preset similarity threshold. If the similarity is greater than the preset similarity threshold, the loss function converges; if the similarity is less than or equal to the preset similarity threshold, the convergence condition has not been met.

[0105] In the above-mentioned speech conversion method, multiple sample text information and sample emotional speech information carrying different emotional states corresponding to each sample text information are obtained, text analysis processing is performed on each sample text information to obtain multiple labeled sample text information, parameter extraction is performed on each sample emotional speech information to obtain the acoustic parameters of each sample emotional speech information, and an initial emotional speech acoustic model is trained based on the acoustic parameters of each labeled sample text information and each sample emotional speech information until the preset convergence conditions are met, and the convergence of the initial emotional speech acoustic model is determined to obtain a more accurate emotional speech acoustic model.

[0106] In one embodiment, in order to facilitate understanding by those skilled in the art, the voice conversion method is described in detail below. Figure 9 As shown, the method may include:

[0107] S901, obtaining the facial image and body key point position information of the speaker;

[0108] S902, extracting the facial expression of the speaker through the facial image, and determining the body movements of the speaker through the position information of the key points of the body;

[0109] S903, determining the emotional state of the sender based on facial expressions and body movements;

[0110] S904, obtaining an emotion label corresponding to the emotional state;

[0111] S905, matching the emotion label with the label in each emotional speech acoustic model, and determining the emotional speech acoustic model that is successfully matched as the emotional speech acoustic model;

[0112] S906, extracting keyword text from the target voice information;

[0113] S907, performing search analysis on the keyword text to obtain multiple candidate search text information;

[0114] S908, performing word fusion processing on the keyword text according to the search text information with the highest similarity to the keyword text among the candidate search text information to obtain text information of the target voice information;

[0115] S909: Input the text information into the emotional speech acoustic model to obtain emotional speech information corresponding to the target speech information.

[0116] It should be noted that for the descriptions in S901-S909 above, reference may be made to the relevant descriptions in the above embodiments, and the effects are similar, so this embodiment will not be repeated here.

[0117] Furthermore, it is understandable that Figure 10 This flowchart illustrates the speech conversion method. Taking whispering as an example, the method involves acquiring an image of the whisperer's facial expression, preprocessing the image to extract features, identifying the speaker's emotion when whispering, and selecting an emotional speech acoustic model corresponding to the emotion from multiple emotional speech acoustic models. The whispered speech is then processed, converting it into whisper keyword text. This whisper text is then analyzed and context-sensitively annotated to produce whisper text. This whisper text is then fed into an emotional speech acoustic model for training, yielding the emotional speech information corresponding to the whispered speech.

[0118] In the above-mentioned speech conversion method, the method obtains the facial image and body key point position information of the speaker, then extracts the facial expression of the speaker through the facial image, determines the body movement of the speaker through the body key point position information, and then determines the emotional state of the speaker based on the facial expression and body movement, obtains the emotion label corresponding to the emotional state, matches the emotion label with the label in each emotional speech acoustic model, and finally determines the successfully matched emotional speech acoustic model as the emotional speech acoustic model, extracts the keyword text in the target speech information, performs search analysis on the keyword text, obtains multiple candidate search text information, performs word fusion processing on the keyword text according to the search text information with the highest similarity between each candidate search text information and the keyword text, obtains the text information of the target speech information, inputs the text information into the emotional speech acoustic model, and obtains the emotional speech information corresponding to the target speech information. This method can accurately determine the emotional state of the speaker when sending the target voice information through the speaker's body language, accurately select the emotional speech acoustic model corresponding to the speaker's emotional state from multiple emotional speech acoustic models, and perform voice conversion processing on the target voice information. Compared with the traditional technology of converting the target voice into the voice emitted by the robot, this method can make the voice heard by the receiver be normal voice with emotion, and can enable the receiver to correctly understand the meaning of the speaker's whispered expression, thereby improving the communication efficiency between the speaker and the receiver.

[0119] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0120] Based on the same inventive concept, embodiments of the present application also provide a speech conversion device for implementing the aforementioned speech conversion method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more speech conversion device embodiments provided below can be found in the aforementioned limitations of the speech conversion method and will not be further elaborated here.

[0121] In one embodiment, Figure 11As shown, a speech conversion device is provided, comprising: a first acquisition module 11, a first determination module 12 and a first processing module 13, wherein:

[0122] The first acquisition module 11 is used to acquire the body language and target voice information of the speaker; the target voice information refers to the voice information sent through a specific voice state;

[0123] A first determining module 12 is used to determine the emotional state of the sender when sending the target voice message based on body language;

[0124] The first processing module 13 is used to perform speech conversion processing on the target speech information using the emotional speech acoustic model corresponding to the emotional state to obtain emotional speech information corresponding to the target speech information; the emotional speech information represents speech carrying an emotional state.

[0125] The speech conversion device provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0126] In one embodiment, the first acquisition module includes: an acquisition unit, a first determination unit, and a second determination unit, wherein:

[0127] An acquisition unit, used to acquire the facial image and body key point position information of the speaker;

[0128] The first determining unit is configured to extract the facial expression of the speaker through the facial image and determine the body movements of the speaker through the position information of key body points;

[0129] The second determining unit is used to determine the emotional state of the speaker based on facial expressions and body movements.

[0130] The speech conversion device provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0131] In one embodiment, the voice conversion device further includes: a second acquisition module and a matching module, wherein:

[0132] The second acquisition module is used to obtain the emotion label corresponding to the emotional state;

[0133] The matching module is used to match the emotion label with the label in each emotional speech acoustic model, and determine the emotional speech acoustic model that is successfully matched as the emotional speech acoustic model.

[0134] The speech conversion device provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0135] In one embodiment, the first processing module includes: a third determining unit and a fourth determining unit, wherein:

[0136] A third determining unit is used to extract text information from the target voice information to obtain text information of the target voice information;

[0137] The fourth determining unit is used to input the text information into the emotional speech acoustic model to obtain emotional speech information corresponding to the target speech information.

[0138] The speech conversion device provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0139] Optionally, the third determining unit is specifically configured to extract keyword text from the target voice information; perform word expansion processing on the keyword text to generate text information of the target voice information.

[0140] The speech conversion device provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0141] Optionally, the third determination unit is specifically used to perform search analysis on the keyword text to obtain multiple candidate search text information; based on the search text information with the highest similarity to the keyword text in each candidate search text information, perform word fusion processing on the keyword text to obtain text information of the target voice information.

[0142] The speech conversion device provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0143] In one embodiment, the speech conversion device further includes: a third acquisition module, a second processing module, and a second determination module, wherein:

[0144] The third acquisition module is used to obtain multiple sample text information and sample emotional speech information carrying different emotional states corresponding to each sample text information;

[0145] The second processing module is used to perform text analysis on each sample text information to obtain multiple labeled sample text information, and to extract parameters of each sample emotional speech information to obtain acoustic parameters of each sample emotional speech information;

[0146] The second determination unit is used to train the initial emotional speech acoustic model according to the acoustic parameters of each labeled sample text information and each sample emotional speech information until a preset convergence condition is met, determine that the initial emotional speech acoustic model converges, and obtain the emotional speech acoustic model.

[0147] Optionally, the convergence condition is that the similarity between the emotional speech information output by the initial emotional speech acoustic model and the corresponding sample emotional speech information is greater than a preset similarity threshold.

[0148] The speech conversion device provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0149] Each module in the above-mentioned speech conversion device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0150] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 1 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store speech conversion data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a speech conversion method.

[0151] Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0152] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements all embodiments of the above method steps when executing the computer program.

[0153] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, all embodiments of the above method steps are implemented.

[0154] In one embodiment, a computer program product is provided, comprising a computer program, which implements all embodiments of the above method steps when executed by a processor.

[0155] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0156] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0157] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0158] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A voice conversion method, characterized in that: The method comprises: Acquiring the body language and target voice information of the speaker; the target voice information represents the voice information sent by whispering; determining the emotional state of the speaker when the target voice message is sent based on the body language; The target speech information is divided into non-overlapping frames through the emotional speech acoustic model corresponding to the emotional state, and the target speech information is pre-amplified and filtered, and linear prediction analysis is performed on the filtered target speech information. The predicted target speech information is gain adjusted according to the unvoiced and voiced sound types of the frame to obtain the emotional speech information corresponding to the target speech information; the emotional speech information represents the speech carrying the emotional state.

2. The method according to claim 1, characterized in that The body language includes body movements and facial expressions; and determining the emotional state of the sender when sending the target voice message based on the body language includes: Obtaining the facial image and body key point position information of the speaker; Extracting the facial expression of the speaker through the facial image, and determining the body movements of the speaker through the body key point position information; The emotional state of the speaker is determined based on the facial expression and the body movement.

3. The method according to claim 1 or 2, characterized in that The method further comprises: Obtaining an emotion label corresponding to the emotional state; The emotion label is matched with the label in each emotional speech acoustic model, and the emotional speech acoustic model that is successfully matched is determined as the emotional speech acoustic model.

4. The method according to claim 1 or 2, characterized in that The performing voice conversion processing on the target voice information to obtain emotional voice information corresponding to the target voice information includes: Extracting text information from the target voice information to obtain text information of the target voice information; The text information is input into the emotional speech acoustic model to obtain emotional speech information corresponding to the target speech information.

5. The method according to claim 4, characterized in that The extracting text information from the target voice information to obtain text information of the target voice information includes: Extracting keyword text from the target voice information; Perform word expansion processing on the keyword text to generate text information of the target voice information.

6. The method according to claim 5, characterized in that The step of performing word expansion processing on the keyword text to generate text information of the target voice information includes: Performing search analysis on the keyword text to obtain multiple candidate search text information; According to the retrieval text information with the highest similarity to the keyword text among the candidate retrieval text information, the keyword text is subjected to a word fusion process to obtain the text information of the target voice information.

7. The method according to claim 1 or 2, characterized in that The construction process of the emotional speech acoustic model includes: Acquire multiple sample text information and sample emotional speech information with different emotional states corresponding to each of the sample text information; Performing text analysis on each of the sample text information to obtain a plurality of annotated sample text information, and performing parameter extraction on each of the sample emotional speech information to obtain acoustic parameters of each of the sample emotional speech information; According to the acoustic parameters of each of the labeled sample text information and each of the sample emotional speech information, an initial emotional speech acoustic model is trained until a preset convergence condition is met, and it is determined that the initial emotional speech acoustic model converges to obtain the emotional speech acoustic model.

8. The method according to claim 7, characterized in that The convergence condition is that the similarity between the emotional speech information output by the initial emotional speech acoustic model and the corresponding sample emotional speech information is greater than a preset similarity threshold.

9. A voice conversion device, characterized in that: The device comprises: The first acquisition module is used to acquire the body language of the speaker and target voice information; the target voice information is the voice information sent by whispering; A first determining module is used to determine the emotional state of the speaker when sending the target voice message based on the body language; The first processing module is used to divide the target speech information into non-overlapping frames through the emotional speech acoustic model corresponding to the emotional state, perform pre-amplification filtering on the target speech information, perform linear prediction analysis on the filtered target speech information, and adjust the gain of the predicted target speech information according to the unvoiced and voiced sound types of the frame to obtain emotional speech information corresponding to the target speech information; the emotional speech information represents speech carrying the emotional state.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images