Emotion audio processing method and device, electronic equipment and storage medium

Abstract

The invention provides an emotion audio processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an emotion audio of a user, carrying out the emotion recognition of the emotion audio, and obtaining a target emotion corresponding to the emotion audio; obtaining an animal audio of which the emotion is consistent with the target emotion; and converting the timbre of the animal audio into the timbre of the emotion audio to obtain a target audio. According to the method and device, the electronic equipment and the storage medium provided by the invention, the target audio which carries the timbre and emotion of the user and is represented in the form of animal sound can be obtained, the target audio is played, so that the animal can easily understand the emotion of the user, long-term training and learning for the animal are not needed, and the user experience is improved. Effectiveness of interaction between the user and the animal through the target audio is greatly improved, and interestingness of interaction between the user and the animal is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an emotional audio processing method, device, electronic device and storage medium. Background Art

[0002] The emergence of devices such as pet language translators and pet nannies provides people with a way to understand pets' emotions through their calls. On the contrary, it is difficult for pets to understand human emotions through human voices.

[0003] Currently, although some devices can play recorded sounds to pets, these sounds cannot reflect actual human emotions. Secondly, these sounds are significantly different from the natural voices of pet owners. It is difficult for pets to associate the played sounds with their owners, which makes mechanical sound playback unable to achieve the effect of human-pet interaction. Summary of the invention

[0004] The present invention provides an emotional audio processing method, device, electronic device and storage medium, which are used to solve the defect in the related art that playing recorded calls cannot achieve human-pet interaction.

[0005] The present invention provides an emotional audio processing method, comprising: Acquire the user's emotional audio, perform emotion recognition on the emotional audio, and obtain a target emotion corresponding to the emotional audio; Acquire audio of an animal whose emotion is consistent with the target emotion; The timbre of the animal audio is converted into the timbre of the emotional audio to obtain the target audio.

[0006] According to an emotional audio processing method provided by the present invention, the step of obtaining the emotional audio of a user comprises: Get the user's voice clip; Based on a first background model, emotion detection is performed on the speech segment to obtain a first detection result, wherein the first background model is constructed based on non-specific emotional speech; In the case where the first detection result contains emotion, based on a second background model, performing emotion detection on the speech segment to obtain a second detection result, wherein the second background model is constructed based on the sample emotional speech; When the second detection result is that the voice segment contains emotion, the voice segment is determined to be the emotional audio.

[0007] According to an emotional audio processing method provided by the present invention, the emotion detection is performed on the speech segment based on the second background model to obtain a second detection result, including: Based on a second background model corresponding to the user type of the user, emotion detection is performed on the voice segment to obtain a second detection result, wherein the second background model is constructed based on sample emotional speech under the user type.

[0008] According to an emotional audio processing method provided by the present invention, the emotional audio is subjected to emotion recognition, comprising: Based on an emotion recognition model corresponding to the user type of the user, emotion recognition is performed on the emotion audio, wherein the emotion recognition model is constructed based on acoustic features of sample emotion audio under the user type, and the acoustic features correspond to the user type.

[0009] According to an emotional audio processing method provided by the present invention, the emotional audio is subjected to emotion recognition, further comprising: Emotion recognition is performed on the emotion audio based on a universal emotion recognition model, where the universal emotion recognition model is constructed based on acoustic features of sample emotion audio under various user types.

[0010] According to an emotional audio processing method provided by the present invention, the step of obtaining animal audio whose emotion is consistent with the target emotion comprises: Get the specified animal type of the user; From the candidate animal audios under the specified animal type, the candidate animal audios whose emotions are consistent with the target emotions are selected as the animal audios.

[0011] The present invention also provides an emotional audio processing device, comprising: An emotion recognition unit, used to obtain the user's emotion audio, perform emotion recognition on the emotion audio, and obtain a target emotion corresponding to the emotion audio; An audio acquisition unit, used to acquire audio of an animal whose emotion is consistent with the target emotion; The timbre conversion unit is used to convert the timbre of the animal audio into the timbre of the emotional audio to obtain the target audio.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned emotional audio processing methods is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the emotional audio processing method described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the emotional audio processing method described above is implemented.

[0015] The emotional audio processing method, device, electronic device and storage medium provided by the present invention perform emotion recognition on the user's emotional audio to obtain the target emotion, and then convert the timbre of the animal audio with the same emotion as the target emotion into the timbre of the emotional audio. The target audio thus obtained carries the user's timbre and emotion and is in the form of animal calls. Playing the target audio can make the animal easily understand the user's emotions without the need for long-term training and learning for the animals. This greatly improves the effectiveness of realizing interaction between users and animals through target audio, and helps to enhance the fun of human-pet interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the emotional audio processing method provided by the present invention.

[0018] Figure 2 It is a structural schematic diagram of the emotional audio processing device provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Effective communication between pets and their owners has always been a key factor that restricts pet owners from enjoying more fun with their pets. Due to the limitations of communication methods, owners often find it difficult to accurately convey their emotions to their pets, resulting in pets being unable to fully understand their owners' intentions and emotions, thus affecting the interactive experience between owners and pets.

[0022] In order to solve this problem, devices such as pet language translators and pet nanny machines have emerged. These devices mainly recognize pet emotions by analyzing the pet's barking. Taking pet dogs as an example, these devices will store various possible barking sound samples in the database in advance according to the dog's breed, size, body shape, nose and other characteristics. In actual use, the device will collect the dog's barking and compare it with the sound samples in the database to determine the dog's current emotional state, such as happy, sad or angry, and give corresponding prompt information. This method can help the owner better understand the pet's emotions. But on the other hand, it is difficult for pets to accurately judge the owner's emotions in a similar way.

[0023] Although pets can distinguish the timbre of different people's speech, it takes long-term training and learning to distinguish specific emotional expressions from voices. Although there are currently solutions to play recorded calls to pets, or to mechanically generate and play calls, these calls cannot reflect the owner's true emotions at the moment, and these calls are significantly different from the natural voice of the pet owner. It is difficult for pets to associate the played calls with their owners, which makes mechanical call playback unable to achieve the effect of human-pet interaction.

[0024] In order to solve the above problems, the present invention provides an emotional audio processing method. Figure 1 is a flow chart of the emotional audio processing method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, obtaining the user's emotional audio, performing emotion recognition on the emotional audio, and obtaining a target emotion corresponding to the emotional audio.

[0025] Specifically, the user is a person who needs to interact with the animal, for example, a pet owner. The user's emotional audio is the user's voice containing emotions, and the emotions contained are the emotions that the user hopes to convey to the animal.

[0026] The emotional audio here can be obtained through a sound pickup device or through the Internet. The sound pickup device here can be a smart phone, a tablet computer, or a smart appliance such as a stereo, a television, and an air conditioner. After the sound pickup device obtains the emotional audio through a microphone array, it can also amplify and reduce noise on the emotional audio. The embodiment of the present invention does not specifically limit this.

[0027] Moreover, considering that there are various ways to collect and acquire emotional audio, the encoding formats of different emotional audio may also be different. It is also possible to perform unified sound encoding format conversion on the emotional audio. For example, they can all be converted into 16K16bit linear PCM (Pulse Code Modulation) format to facilitate subsequent audio processing.

[0028] After obtaining the user's emotional audio, emotion recognition can be performed on the emotional audio. Here, the way to perform emotion recognition can be to input the emotional audio into a pre-trained emotion recognition model, and the emotion recognition model performs emotion recognition on the emotional audio. The emotion contained in the emotional audio thus recognized is recorded as the target emotion in the embodiment of the present invention. Here, the emotion recognition model can be constructed based on a classification model, for example, it can be a deep neural network, a convolutional neural network, a support vector machine, etc.

[0029] The target emotion may be a type of emotion, such as happiness, sadness, anger, etc. It can be understood that the target emotion is the information that the user wants to convey when interacting with the animal in the embodiment of the present invention.

[0030] Step 120 , obtaining animal audio whose emotion is consistent with the target emotion.

[0031] Specifically, after obtaining the target emotion corresponding to the emotional audio, animal audio with the same emotion as the target emotion can be selected from pre-stored candidate animal audios that can reflect various emotions, that is, animal audio that can reflect the target emotion can be selected.

[0032] The animal audio thus obtained is a real recorded sound of an animal, and the emotion reflected by the animal audio is consistent with the target emotion reflected by the emotional audio in step 110. It can be understood that, compared with the user's emotional audio, the animal can easily understand the target emotion from the played animal audio without long-term training and learning.

[0033] For example, if the target emotion reflected by the user's emotional audio is happiness, and the animal the user wants to communicate with is a dog, then the audio of a dog barking that reflects happiness can be selected from the real recorded audios of dogs barking that can reflect various emotions as the animal audio.

[0034] Step 130: convert the timbre of the animal audio into the timbre of the emotional audio to obtain the target audio.

[0035] Specifically, the timbre in the animal audio is the timbre of the animal itself. When the animal hears the played animal audio, it cannot associate the emotion reflected in the animal audio with the user itself. Therefore, after obtaining the animal audio, the timbre of the animal audio can be converted, specifically, the timbre of the emotional audio is transplanted into the animal audio, thereby obtaining an audio with the same emotional content and timbre as the animal audio, that is, obtaining the target audio.

[0036] Here, the timbre conversion can be achieved through a pre-trained timbre conversion model, such as Tacotron2, WaveNet, FastSpeech, etc. Specifically, the animal audio and emotional audio can be input into the timbre conversion model together, and the timbre conversion model performs timbre conversion and outputs the target audio.

[0037] In addition, considering that the timbre of humans and animals is quite different, and the pre-trained timbre conversion models are usually aimed at general scenarios, the conversion effect between animal timbre and human timbre based on the pre-trained timbre conversion model may not be good. In this case, fine-tuning can be performed on the basis of the pre-trained timbre conversion model to achieve model optimization, so as to better cope with the timbre conversion task between animal timbre and human timbre and realize rapid cloning of user timbre.

[0038] It is understandable that in the embodiment of the present invention, the target audio obtained by timbre conversion is in the form of animal calls, reflecting the target emotion contained in the user's emotional audio, and the timbre is consistent with the user. Playing the target audio to the animal can make the animal easily understand the emotion from the target audio without long-term training and learning of the animal, and associate the understood emotion with the user through the timbre of the target audio, thereby greatly improving the effectiveness of the interaction between the user and the animal through the target audio.

[0039] In the method provided in the embodiment of the present invention, emotion recognition is performed on the user's emotional audio to obtain the target emotion, and then the timbre of the animal audio with the same emotion as the target emotion is converted into the timbre of the emotional audio. The target audio thus obtained carries the user's timbre and emotion and is in the form of animal calls. Playing the target audio can make the animal easily understand the user's emotions without the need for long-term training and learning for the animals, which greatly improves the effectiveness of interaction between users and animals through target audio and helps to enhance the fun of human-pet interaction.

[0040] Based on the above embodiment, in step 110, obtaining the user's emotional audio includes: Get the user's voice clip; Based on a first background model, emotion detection is performed on the speech segment to obtain a first detection result, wherein the first background model is constructed based on non-specific emotional speech; In the case where the first detection result contains emotion, based on a second background model, performing emotion detection on the speech segment to obtain a second detection result, wherein the second background model is constructed based on the sample emotional speech; When the second detection result is that the voice segment contains emotion, the voice segment is determined to be the emotional audio.

[0041] Specifically, the user's voice segment is the collected user voice. Considering that the original collected audio may contain human voice and other sounds mixed, endpoint detection (Voice Activity Detection, VAD) can be used to identify the active part and the inactive part of the original collected audio, where the active part is the voice segment in the embodiment of the present invention.

[0042] When a user speaks in daily life, some of their voices may express emotions, while others may not contain specific emotions. Therefore, a user's voice clip may be emotional audio containing emotions, or it may be a normal human voice without emotions, that is, non-specific emotional audio. Therefore, after obtaining the user's voice clip, it is necessary to determine whether the voice clip is emotional audio containing emotions.

[0043] When judging whether a speech segment is emotional audio containing emotions, it can be achieved through the first background model and the second background model.

[0044] Among them, the first background model and the second background model are both models for distinguishing whether a speech segment is emotional audio, and the first background model and the second background model can both be models constructed based on a large number of speech segments, for example, they can be Gaussian Mixture Model (GMM) trained by Expectation-Maximization (EM) algorithm.

[0045] The first background model is specifically constructed based on non-specific emotional speech. Here, non-specific emotional speech refers to a sample speech segment that is collected in advance and does not contain emotions. The first background model can be a universal background model (UBM) that characterizes non-specific emotional speech. Since a large amount of non-specific emotional speech is used in the construction of the first background model, the first background model can fit the acoustic feature distribution of non-specific emotional speech. The acoustic features here can be spectral parameter features, specifically Linear Prediction Cepstral Coefficients (LPCC), Mel Frequency Cepstral Coefficients (MFCC), Perceptual Linear Predictive Coding (PLP) features, etc.

[0046] When emotion detection is performed on a speech segment based on the first background model, the similarity between the acoustic features of the first background model and the speech segment can be calculated. If the similarity is greater than a first threshold, it can be determined that the speech segment does not contain emotion. If the similarity is less than or equal to the first threshold, it can be determined that the speech segment contains emotion. Here, the result of emotion detection based on the first background model is recorded as the first detection result.

[0047] The second background model is specifically constructed based on sample emotional speech. Here, the sample emotional speech is a sample speech segment containing emotions collected in advance. The second background model can be a UBM that characterizes the sample emotional speech. Since a large number of sample emotional speech are used to construct the second background model, the second background model can fit the acoustic feature distribution of the sample emotional speech.

[0048] When emotion detection is performed on the speech segment based on the second background model, the similarity between the acoustic features of the second background model and the speech segment can be calculated. If the similarity is greater than a second threshold, it can be determined that the speech segment contains emotion. If the similarity is less than or equal to the second threshold, it can be determined that the speech segment does not contain emotion. Here, the result of emotion detection based on the second background model is recorded as the second detection result.

[0049] Furthermore, in the process of combining the first background model and the second background model to perform emotion detection on a speech clip, the first background model can be first applied to perform emotion detection, and when it is determined that the detection result contains emotion, the second background model can be applied to perform emotion detection, thereby further determining whether the detection result contains emotion, that is, determining whether the speech clip is emotional audio.

[0050] In the method provided in the embodiment of the present invention, the first background model and the second background model are combined to perform reverse and forward emotion detection on the voice segment, thereby ensuring that the acquired emotional audio contains emotions, thereby ensuring the reliability and accuracy of emotion detection.

[0051] Furthermore, by combining the first background model and the second background model, effective preprocessing for speech segments can be achieved, which helps to improve the accuracy of emotion recognition.

[0052] Based on any of the above embodiments, in step 110, emotion detection is performed on the voice segment based on the second background model to obtain a second detection result, including: Based on a second background model corresponding to the user type of the user, emotion detection is performed on the voice segment to obtain a second detection result, wherein the second background model is constructed based on sample emotional speech under the user type.

[0053] Specifically, considering that different users have differences in emotional expression, which is reflected in the different acoustic distributions of emotional speech, different second background models can be constructed for different user types in the embodiment of the present invention.

[0054] That is, multiple second background models can be pre-constructed, each second background model corresponds to a user type, specifically, a second background model is constructed based on sample emotional speech of a user type. For example, multiple user types such as the elderly, children, and middle-aged people can be divided, and a second background model corresponding to the elderly is constructed for the sample emotional speech collected from the elderly, and a second background model corresponding to the children is constructed for the sample emotional speech collected from the children.

[0055] Therefore, after the first detection result obtained by the first background model contains emotions, the user type to which the user belongs can be determined, and the second background model corresponding to the user type is applied to perform emotion detection on the voice segment, thereby obtaining a second detection result. It can be understood that emotion detection based on the second background model corresponding to the user type makes the emotion detection for the voice segment more consistent with the user's own emotional expression, which can effectively improve the reliability and accuracy of emotion detection.

[0056] In the method provided in the embodiment of the present invention, different second background models are constructed for different user types, so that emotion detection can fit the user's own emotional expression mode, thereby improving the reliability of emotion detection.

[0057] Moreover, compared with the sample emotional audio of different user types that needs to be acquired for the construction of the second background model for different user types, the non-specific emotional speech required for the construction of the first background model is obviously easier to acquire and less expensive. Therefore, in the process of emotion detection, emotion detection can be first performed based on the first background model to achieve a rough screening of emotional audio. When it is determined that the first emotion detection result contains emotions, emotion detection is then performed based on the second background model to achieve a fine screening of emotional audio, thereby improving the accuracy of emotion detection.

[0058] Based on any of the above embodiments, in step 110, the step of performing emotion recognition on the emotional audio includes: Based on an emotion recognition model corresponding to the user type of the user, emotion recognition is performed on the emotion audio, wherein the emotion recognition model is constructed based on acoustic features of sample emotion audio under the user type, and the acoustic features correspond to the user type.

[0059] Specifically, considering that different users have differences in emotional expression, different emotion recognition models can be built for different user types. In addition, considering that audio itself is a redundant signal, directly using audio for modeling will lead to an unnecessary increase in the scale of the model. Therefore, acoustic features can be extracted from audio to extract effective information and discard irrelevant redundant information, and the model can be built based on acoustic features.

[0060] That is, multiple emotion recognition models can be pre-selected and constructed, each emotion recognition model corresponds to a user type, specifically, an emotion recognition model is constructed based on the acoustic features of sample emotion audio under a user type and its corresponding emotion label, and when the emotion recognition model performs emotion recognition on the emotion audio under the user type, the type of acoustic features applied to the emotion audio corresponds to the user type. Here, the emotion labels corresponding to the sample emotion audio can be obtained by manual annotation, and for sample emotion audio in different languages, experts familiar with the corresponding languages ​​can be invited to annotate.

[0061] That is to say, for different user types, there may be sample emotional audio for that user type, and the acoustic features corresponding to that user type are extracted from the sample emotional audio for that user type, and the extracted acoustic features are applied to the training of the emotion recognition model corresponding to that user type. During the model training process, the difference between the probability that the emotion estimated by the model for the sample emotional audio belongs to various emotion categories and the labeled emotion label can be used as the loss of model training to guide the iteration of model parameters.

[0062] For example, multiple user types can be divided, such as the elderly, children, and middle-aged people. Among them, children are smaller in size, and their emotional audio is short, fast, and has a lower volume. Therefore, it is suitable to extract features with strong periodicity as the corresponding acoustic features for children, such as fundamental frequency, zero-crossing rate, syllable length, etc.; the emotional audio of middle-aged people is louder and more explosive, so it is suitable to extract features such as loudness and rhythm (the time ratio of explosive sounds to silent sounds in the sound) as the corresponding acoustic features for middle-aged people; the voices of the elderly are low and weak, so it is suitable to extract features including spectral flatness, spectral roll-off coefficient, low-frequency energy coefficient, etc. as the acoustic features for the elderly.

[0063] Therefore, corresponding emotion recognition models can be built for different user types. For example, for children, an emotion recognition model can be built based on acoustic features such as fundamental frequency, zero crossing rate, syllable length, etc. in the sample emotion audio of children; for middle-aged people, an emotion recognition model can be built based on acoustic features such as loudness and rhythm in the sample emotion audio of middle-aged people; for the elderly, an emotion recognition model can be built based on acoustic features such as spectral flatness, spectral roll-off coefficient, low-frequency energy coefficient, etc. in the sample emotion audio of the elderly.

[0064] After obtaining the user's emotional audio and determining the user type to which the user belongs, the acoustic features corresponding to the user type in the emotional audio can be input into the emotion recognition model corresponding to the user type, and the emotion recognition model corresponding to the user type uses the acoustic features corresponding to the user type in the emotional audio to perform emotion recognition, thereby obtaining and outputting the emotion recognition result. The emotion recognition result here can be directly used as the target emotion contained in the emotional audio, or it can be combined with other information to determine the target emotion contained in the emotional audio.

[0065] In the method provided in an embodiment of the present invention, acoustic features for emotion recognition are determined for different user types, and emotion recognition models are modeled based on the acoustic features of different user types, so as to realize personalized emotion recognition of emotional audio for different user types, thereby ensuring the accuracy and reliability of emotion recognition.

[0066] Moreover, the model used for emotion recognition is trained based on a deep learning model, which has the advantages of high recognition efficiency and high accuracy, and is more conducive to the localization and embedding of the emotion audio processing system.

[0067] Based on any of the above embodiments, in step 110, the performing emotion recognition on the emotional audio further includes: Emotion recognition is performed on the emotion audio based on a universal emotion recognition model, where the universal emotion recognition model is constructed based on acoustic features of sample emotion audio under various user types.

[0068] Specifically, in addition to modeling personalized emotion recognition models for different user types, a universal emotion recognition model can also be uniformly modeled for all user types. In an embodiment of the present invention, in order to facilitate the distinction between the emotion recognition models corresponding to the user types mentioned in the above embodiment, the uniformly modeled, universal emotion recognition model can be recorded as a universal emotion recognition model.

[0069] Here, the general emotion recognition model can be trained based on training samples that are a mixture of sample emotion audios of various user types and their corresponding emotion labels. During the model training process, the difference between the probability that the emotion estimated by the model for the sample emotion audio belongs to various emotion categories and the labeled emotion labels can be used as the loss of model training to guide the iteration of model parameters.

[0070] In the process of emotion recognition for emotional audio, in addition to applying the emotion recognition model corresponding to the user type to perform emotion recognition on the emotional audio, a general emotion recognition model can also be applied to perform emotion recognition on the emotional audio. Specifically, the acoustic features of the emotional audio can be input into the general emotion recognition model, and the general emotion recognition model performs emotion recognition on the acoustic features of the input emotional audio. The acoustic features here can be acoustic features common to various user types, such as cepstrum parameters.

[0071] It is understandable that the emotion recognition model corresponding to the user type and the general emotion recognition model can output their own emotion recognition results for the emotion audio respectively. The emotion recognition result here can be the probability that the emotion contained in the emotion audio belongs to each emotion type. After obtaining two emotion recognition results, the probabilities of belonging to each emotion type in the two emotion recognition results can be weighted and summed respectively, thereby obtaining the comprehensive probability that the emotion contained in the emotion audio belongs to each emotion type, and the emotion type corresponding to the maximum value of the comprehensive probability can be selected as the target emotion.

[0072] In the method provided in the embodiment of the present invention, emotional audio can be subjected to emotion recognition by combining personalized emotion recognition models for different user types and a universal emotion recognition model for all user types. Emotion recognition is performed from two aspects respectively, which can effectively ensure the reliability and accuracy of target emotion acquisition.

[0073] Based on any of the above embodiments, in step 120, obtaining the animal audio having an emotion consistent with the target emotion includes: Get the specified animal type of the user; From the candidate animal audios under the specified animal type, the candidate animal audios whose emotions are consistent with the target emotions are selected as the animal audios.

[0074] Specifically, since different types of animals differ in their calls and the emotional expressions in their calls, animal audio of different animal types can be collected separately, which are referred to as candidate animal audios.

[0075] In addition, experts who understand or study the animal type may be invited to perform emotion annotation on the candidate animal audio of the animal type. For example, experts who understand the emotional sounds of dogs may be invited to listen to the pre-collected audio of dog calls, that is, the candidate animal audio of dogs, and then annotate the corresponding emotions for each candidate animal audio, such as "happy", "sad", "angry", etc. Here, different experts may be required to perform emotion annotation for different animal types.

[0076] During the emotional audio processing, the user's designated animal type may be obtained. Here, the designated animal type is the animal type designated by the user and requiring interaction. For example, if the user is a pet owner and the user's pet is a dog, the designated animal type may be a dog; for another example, on the interactive interface of the device, there may be multiple candidate animal types, and the user may select one as the designated animal type, which is not specifically limited in the embodiment of the present invention.

[0077] After determining the specified animal type and the target emotion corresponding to the emotional audio, candidate animal audios under the specified animal type can be selected from the candidate animal audios under various animal types, and candidate animal audios whose marked emotions are consistent with the target emotion can be further selected from the candidate animal audios under the specified animal type as the animal audio.

[0078] It can be understood that the animal audio selected in this way, on the one hand, belongs to the user's specified animal type, and on the other hand, has emotions consistent with the target emotions reflected by the user's emotional audio, and can better convey the target emotions to the animals under the user's specified animal type.

[0079] Based on any of the above embodiments, an embodiment of the present invention provides an emotional audio processing method, including: First, collect the audio and preprocess it: A large amount of audio can be collected from the Internet. Considering the complexity of audio sources and the uneven quality, the collected audio needs to be preprocessed. The preprocessing operation here includes unified audio coding format conversion for audio, as well as endpoint detection and emotion detection for audio.

[0080] Here, the implementation of endpoint detection and emotion detection for audio includes: first, endpoint detection is performed on the audio to extract silent audio segments and retain voice segments. Subsequently, the acoustic features of the voice segments are extracted and compared with the common first background model for similarity, and a threshold judgment is performed on the similarity to eliminate voice segments that do not contain emotions. Then, according to the user type corresponding to the remaining voice segments, the acoustic features of the remaining voice segments are compared with the second background model under the user type, and a threshold judgment is performed on the similarity, thereby obtaining voice segments containing emotions as sample emotional audio.

[0081] Accordingly, the emotional audio of the user that needs to be processed can be obtained through endpoint detection and emotion detection.

[0082] Secondly, emotion recognition modeling: For the sample emotional audio obtained in the previous step, the emotional category can be labeled.

[0083] Furthermore, based on the user type corresponding to the sample emotional audio, acoustic features corresponding to the user type can be extracted from the sample emotional audio, thereby modeling an emotion recognition model for the user type; in addition, cepstrum parameters can be extracted from the sample emotional audio to model a general emotion recognition model.

[0084] After obtaining the user's emotional audio, the emotion recognition model for the user type and the general emotion recognition model may be combined to recognize the target emotion in the emotional audio.

[0085] Next, create an animal emotion sound database: The sounds of various types of animals can be collected online as candidate animal audios, and experts can be invited to label the emotions of the candidate animal audios. After the labeling is completed, a corresponding emotional sound database can be established for different animal types. For example, for dogs, the corresponding emotional sound database can store candidate animal audios of dogs and their emotional labels.

[0086] After the target emotion is identified, the animal audio with the same emotion as the target emotion can be selected from the corresponding emotional sound database in combination with the animal type specified by the user.

[0087] Finally, clone the user patch: The user's voice cloning can be achieved through a pre-trained voice conversion model; or the pre-trained voice conversion model can be fine-tuned, and then the fine-tuned voice conversion model can be applied to clone the voice reflected by the user's emotional audio into the animal audio, thereby obtaining a target audio whose emotional content is consistent with the animal audio and whose voice is consistent with the emotional audio.

[0088] For example, if the owner wants to express anger to his pet husky dog, he can record an angry voice as emotional audio, and decode the emotional audio through the general emotion recognition model and the emotion recognition model for the user type to which the owner belongs, to determine that the target emotion carried in the emotional audio is anger. Then, a husky dog's angry barking sound is found in the husky dog's emotional sound database as action audio, and finally the timbre of the animal audio is converted into the timbre of the owner in the emotional audio through timbre cloning technology and played.

[0089] The emotional audio processing device provided by the present invention is described below. The emotional audio processing device described below and the emotional audio processing method described above can be referenced to each other.

[0090] Figure 2 is a schematic diagram of the structure of the emotional audio processing device provided by the present invention, such as Figure 2 As shown, the device comprises: The emotion recognition unit 210 is used to obtain the user's emotion audio, perform emotion recognition on the emotion audio, and obtain the target emotion corresponding to the emotion audio; An audio acquisition unit 220 is used to acquire animal audio whose emotion is consistent with the target emotion; The timbre conversion unit 230 is used to convert the timbre of the animal audio into the timbre of the emotion audio to obtain the target audio.

[0091] In the device provided in the embodiment of the present invention, emotion recognition is performed on the user's emotional audio to obtain the target emotion, and then the timbre of the animal audio with the same emotion as the target emotion is converted into the timbre of the emotional audio. The target audio thus obtained carries the user's timbre and emotion and is in the form of animal calls. Playing the target audio can make the animal easily understand the user's emotions without the need for long-term training and learning for the animals. This greatly improves the effectiveness of interaction between users and animals through target audio, and helps to enhance the fun of human-pet interaction.

[0092] Based on any of the above embodiments, the emotion recognition unit is specifically used for: Get the user's voice clip; Based on a first background model, emotion detection is performed on the speech segment to obtain a first detection result, wherein the first background model is constructed based on non-specific emotional speech; In the case where the first detection result contains emotion, based on a second background model, performing emotion detection on the speech segment to obtain a second detection result, wherein the second background model is constructed based on the sample emotional speech; When the second detection result is that the voice segment contains emotion, the voice segment is determined to be the emotional audio.

[0093] Based on any of the above embodiments, the emotion recognition unit is specifically used for: Based on a second background model corresponding to the user type of the user, emotion detection is performed on the voice segment to obtain a second detection result, wherein the second background model is constructed based on sample emotional speech under the user type.

[0094] Based on any of the above embodiments, the emotion recognition unit is specifically used for: Based on an emotion recognition model corresponding to the user type of the user, emotion recognition is performed on the emotion audio, wherein the emotion recognition model is constructed based on acoustic features of sample emotion audio under the user type, and the acoustic features correspond to the user type.

[0095] Based on any of the above embodiments, the emotion recognition unit is further used for: Emotion recognition is performed on the emotion audio based on a universal emotion recognition model, where the universal emotion recognition model is constructed based on acoustic features of sample emotion audio under various user types.

[0096] Based on any of the above embodiments, the audio acquisition unit is specifically used for: Get the specified animal type of the user; From the candidate animal audios under the specified animal type, the candidate animal audios whose emotions are consistent with the target emotions are selected as the animal audios.

[0097] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the emotional audio processing method, which includes: Acquire the user's emotional audio, perform emotion recognition on the emotional audio, and obtain a target emotion corresponding to the emotional audio; Acquire audio of an animal whose emotion is consistent with the target emotion; The timbre of the animal audio is converted into the timbre of the emotional audio to obtain the target audio.

[0098] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0099] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the emotional audio processing method provided by the above methods, the method includes: Acquire the user's emotional audio, perform emotion recognition on the emotional audio, and obtain a target emotion corresponding to the emotional audio; Acquire audio of an animal whose emotion is consistent with the target emotion; The timbre of the animal audio is converted into the timbre of the emotional audio to obtain the target audio.

[0100] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the emotional audio processing method provided by the above methods, the method comprising: Acquire the user's emotional audio, perform emotion recognition on the emotional audio, and obtain a target emotion corresponding to the emotional audio; Acquire audio of an animal whose emotion is consistent with the target emotion; The timbre of the animal audio is converted into the timbre of the emotional audio to obtain the target audio.

[0101] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0102] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An emotional audio processing method, characterized in that: include: Acquire the user's emotional audio, perform emotion recognition on the emotional audio, and obtain a target emotion corresponding to the emotional audio; Acquire audio of an animal whose emotion is consistent with the target emotion; The timbre of the animal audio is converted into the timbre of the emotional audio to obtain the target audio.

2. The emotional audio processing method according to claim 1, characterized in that: The step of obtaining the user's emotional audio includes: Get the user's voice clip; Based on a first background model, emotion detection is performed on the speech segment to obtain a first detection result, wherein the first background model is constructed based on non-specific emotional speech; In the case where the first detection result contains emotion, based on a second background model, performing emotion detection on the speech segment to obtain a second detection result, wherein the second background model is constructed based on the sample emotional speech; When the second detection result is that the voice segment contains emotion, the voice segment is determined to be the emotional audio.

3. The emotional audio processing method according to claim 2, characterized in that: The performing emotion detection on the voice segment based on the second background model to obtain a second detection result includes: Based on a second background model corresponding to the user type of the user, emotion detection is performed on the voice segment to obtain a second detection result, wherein the second background model is constructed based on sample emotional speech under the user type.

4. The emotional audio processing method according to claim 1, characterized in that: The performing emotion recognition on the emotional audio includes: Based on an emotion recognition model corresponding to the user type of the user, emotion recognition is performed on the emotion audio, wherein the emotion recognition model is constructed based on acoustic features of sample emotion audio under the user type, and the acoustic features correspond to the user type.

5. The emotional audio processing method according to claim 4, characterized in that: The performing emotion recognition on the emotional audio further includes: Emotion recognition is performed on the emotion audio based on a universal emotion recognition model, where the universal emotion recognition model is constructed based on acoustic features of sample emotion audio under various user types.

6. The emotional audio processing method according to any one of claims 1 to 5, characterized in that: The step of obtaining the animal audio whose emotion is consistent with the target emotion comprises: Get the specified animal type of the user; From the candidate animal audios under the specified animal type, the candidate animal audios whose emotions are consistent with the target emotions are selected as the animal audios.

7. An emotional audio processing device, characterized in that: include: An emotion recognition unit, used to obtain the user's emotion audio, perform emotion recognition on the emotion audio, and obtain a target emotion corresponding to the emotion audio; An audio acquisition unit, used to acquire audio of an animal whose emotion is consistent with the target emotion; The timbre conversion unit is used to convert the timbre of the animal audio into the timbre of the emotional audio to obtain the target audio.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the emotional audio processing method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the emotional audio processing method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the emotional audio processing method according to any one of claims 1 to 6 is implemented.

Images