Voice interaction method and device, equipment and storage medium

Abstract

The invention provides a voice interaction method and device, equipment and a storage medium, and the method comprises the steps: responding to a voice pacifying request triggered by a target object, and collecting the interaction voice data and object image data of the target object; determining emotion recognition data of the target object according to interactive voice features corresponding to the interactive voice data and image emotion data corresponding to the object image data; and executing target pacifying operation according to the emotion recognition data and the emotion pacifying strategy corresponding to the emotion recognition data to obtain an emotion pacifying result. The probability of angry moods of the user during voice interaction is reduced, and the voice interaction experience and satisfaction degree of the user are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a voice interaction method, apparatus, device and storage medium. Background Art

[0002] At present, with the rapid development of smart terminal technology such as smart TV and voice recognition technology, more and more smart terminals such as smart TV are integrated with voice assistants, so that users can interact with smart terminals such as smart TV through voice, thereby optimizing the interaction scenarios of smart terminals such as smart TV. However, in the existing voice interaction process, the voice assistant may recognize the content incorrectly for many times or fail to query the resources required by the user, causing the user to have emotional fluctuations, and after the user has emotional fluctuations, the voice assistant cannot make corresponding feedback operations, causing the user to have more drastic emotional fluctuations, resulting in a poor terminal experience for the user. Summary of the invention

[0003] The embodiments of the present application provide a voice interaction method, apparatus, device and storage medium, which are intended to solve the technical problem in the prior art that corresponding feedback operations cannot be performed according to user emotions during voice interaction with a smart terminal, resulting in a poor voice interaction experience.

[0004] On the one hand, an embodiment of the present application provides a voice interaction method, the voice interaction method comprising the following steps:

[0005] In response to a voice soothing request triggered by a target object, collecting interactive voice data and object image data of the target object;

[0006] The emotion data determines the emotion recognition data of the target object according to the interactive speech features corresponding to the interactive speech data and the image emotion data corresponding to the object image data;

[0007] A target soothing operation is performed according to the emotion recognition data and an emotion soothing strategy corresponding to the emotion recognition data to obtain an emotion soothing result.

[0008] In a possible implementation of the present application, the emotion data determines the emotion recognition data of the target object according to the interactive voice features corresponding to the interactive voice data and the image emotion data corresponding to the object image data, including:

[0009] Performing emotion classification on the interactive speech features corresponding to the interactive speech data to obtain speech emotion data corresponding to the interactive speech features;

[0010] Performing facial expression recognition on the object image features corresponding to the object image data to obtain image emotion data corresponding to the object image data;

[0011] Matching is performed based on the voice emotion data and the image emotion data to obtain emotion recognition data of the target object.

[0012] In a possible implementation of the present application, the emotion data determines the emotion recognition data of the target object according to the interactive voice features corresponding to the interactive voice data and the image emotion data corresponding to the object image data, including:

[0013] Performing emotion classification on the interactive speech features corresponding to the interactive speech data to obtain speech emotion data corresponding to the interactive speech features;

[0014] Performing facial expression recognition on the object image features corresponding to the object image data to obtain image emotion data corresponding to the object image data;

[0015] Matching is performed based on the voice emotion data and the image emotion data to obtain emotion recognition data of the target object.

[0016] In a possible implementation of the present application, the emotion classification of the interactive speech feature corresponding to the interactive speech data to obtain the speech emotion data corresponding to the interactive speech feature includes:

[0017] Extracting speech features from the interactive speech data to obtain interactive speech features of the interactive speech data;

[0018] Classifying the initial emotion of the interactive speech feature to obtain the initial emotion feature of the interactive speech feature;

[0019] If the initial emotional feature is a negative emotional feature, a secondary emotional classification is performed on the initial emotional feature according to a preset speech recognition model to obtain speech emotion data corresponding to the interactive speech feature.

[0020] In a possible implementation of the present application, performing expression recognition on the object image features corresponding to the object image data to obtain the image emotion data corresponding to the object image data includes:

[0021] Extracting facial features from the object image data to obtain object image features corresponding to the object image data;

[0022] According to the target convolution module, the random gradient descent module and the target feature extractor, expression recognition is performed on the object image features to obtain image emotion data corresponding to the object image data.

[0023] In a possible implementation of the present application, matching the voice emotion data and the image emotion data to obtain the emotion recognition data of the target object includes:

[0024] Acquire the voice emotion level corresponding to the voice emotion data, and acquire the image emotion level corresponding to the image emotion data;

[0025] If the difference between the speech emotion level and the image emotion level is less than or equal to a preset first difference threshold, the speech emotion level or the image emotion level with a larger value among the speech emotion level and the image emotion level is used as the emotion recognition data of the target object;

[0026] If the level difference between the voice emotion level and the image emotion level is greater than or equal to a preset second difference threshold, the speech recognition model is used to perform secondary recognition on the interactive speech data and the interactive speech data to obtain emotion recognition data of the target object.

[0027] In a possible implementation of the present application, the using of a speech recognition model to perform secondary recognition on the interactive speech data and the interactive speech data to obtain emotion recognition data of the target object includes:

[0028] Performing secondary recognition on the interactive voice data and the interactive voice data using a voice recognition model to obtain an updated emotion level, wherein the updated emotion level includes an updated voice emotion level and / or an updated image emotion level;

[0029] The speech emotion level, the image emotion level, the average value of the updated speech emotion level and / or the updated image emotion level are calculated to obtain the emotion recognition data of the target object.

[0030] In a possible implementation of the present application, performing a target soothing operation according to the emotion recognition data and an emotion soothing strategy corresponding to the emotion recognition data to obtain an emotion soothing result includes:

[0031] Obtaining a target emotion level in the emotion recognition data, and generating an emotion soothing measurement strategy corresponding to the target emotion level;

[0032] Outputting corresponding emotion soothing content based on the emotion soothing strategy to obtain an emotion soothing result.

[0033] In a possible implementation of the present application, outputting corresponding emotion soothing content based on the emotion soothing strategy to obtain an emotion soothing result includes:

[0034] Accessing a cloud database to obtain the emotion soothing strategy and the emotion soothing content corresponding to the interactive voice data;

[0035] Outputting the emotion soothing content to the target object and collecting emotion update data of the target object;

[0036] The emotion update data and the emotion recognition data are compared to obtain an emotion soothing result of the target object.

[0037] On the other hand, the present application provides a voice interaction device, the voice interaction device comprising:

[0038] A data collection module, configured to respond to a voice soothing request triggered by a target object and collect interactive voice data and object image data of the target object;

[0039] An emotion recognition module is configured to determine emotion recognition data of the target object according to the interactive voice features corresponding to the interactive voice data and the image emotion data corresponding to the object image data;

[0040] The voice interaction module is configured to perform a target soothing operation according to the emotion recognition data and an emotion soothing strategy corresponding to the emotion recognition data to obtain an emotion soothing result.

[0041] On the other hand, the present application also provides a voice interaction device, the voice interaction device comprising:

[0042] one or more processors;

[0043] Memory; and

[0044] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the steps of the voice interaction method.

[0045] On the other hand, the present application also provides a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the steps in the voice interaction method.

[0046] In this application, by responding to the voice soothing request triggered by the target object, the interactive voice data and object image data of the target object are collected; the emotional data determines the emotional recognition data of the target object according to the interactive voice features corresponding to the interactive voice data and the image emotional data corresponding to the object image data; the target soothing operation is performed according to the emotional recognition data and the emotional soothing strategy corresponding to the emotional recognition data to obtain the emotional soothing result. The emotional recognition data of the target object in the voice interaction process is obtained, and then the emotion recognition is performed in multiple dimensions according to the interactive voice data and the object image data, so as to determine the emotional recognition data of the target object in the voice interaction process, and the corresponding soothing operation is performed according to the emotional soothing strategy corresponding to the emotional mouse data, thereby reducing the probability of the user being angry during the voice interaction and improving the user's voice interaction experience and satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 A schematic diagram of a scenario of a voice interaction method according to an embodiment of the present application;

[0049] Figure 2 A flowchart of an embodiment of a voice interaction method in an embodiment of the present application;

[0050] Figure 3 A flowchart of an embodiment of image emotion data recognition in the voice interaction method provided in the embodiment of the present application;

[0051] Figure 4 A flowchart of an embodiment of performing emotion matching and determining emotion recognition data of a target object in a voice interaction method provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of the structure of an embodiment of a voice interaction device provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of the structure of an embodiment of a voice interaction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0055] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0056] In this application, the word "exemplary" is used to mean "serving as an example, illustration, or illustration." Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any person skilled in the art to implement and use the invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the invention can be implemented without using these specific details. In other instances, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in this application.

[0057] At present, with the rapid development of smart terminal technology such as smart TV and voice recognition technology, more and more smart terminals such as smart TV are integrated with voice assistants, so that users can interact with smart terminals such as smart TV through voice, thereby optimizing the interaction scenarios of smart terminals such as smart TV. However, in the existing voice interaction process, the voice assistant may recognize the content incorrectly for many times or fail to query the resources required by the user, causing the user to have emotional fluctuations, and after the user has emotional fluctuations, the voice assistant cannot make corresponding feedback operations, causing the user to have more drastic emotional fluctuations, resulting in a poor terminal experience for the user.

[0058] Based on this, the present application proposes a voice interaction method, device, equipment and computer-readable storage medium to solve the technical problem in the prior art that corresponding feedback operations cannot be performed according to user emotions when performing voice interaction with a smart terminal, resulting in a poor voice interaction experience.

[0059] The voice interaction method in the embodiment of the present invention is applied to a voice interaction device, and the voice interaction device is arranged in a voice interaction device. The voice interaction device is provided with one or more processors, a memory, and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the voice interaction method; wherein the voice interaction device can be an intelligent terminal, such as a mobile phone, a tablet computer, a network device, and a smart computer; optionally, the voice interaction device can also be a server, or a service cluster composed of multiple servers.

[0060] like Figure 1 As shown, Figure 1 This is a scenario diagram of the voice interaction method in an embodiment of the present application. The voice interaction scenario in the embodiment of the present invention includes a voice interaction device 100 (a voice interaction device is integrated in the voice interaction device 100), and a computer-readable storage medium corresponding to the voice interaction method is running in the voice interaction device 100 to execute the steps of the voice interaction method.

[0061] Understandably, Figure 1 The voice interaction devices in the voice interaction method scenario shown, or the devices included in the voice interaction devices, do not constitute limitations on the embodiments of the present invention, that is, the number of devices and types of devices of the voice interaction devices included in the voice interaction method scenario, or the number of devices and types of devices included in each device, do not affect the overall implementation of the technical solution in the embodiments of the present invention, and can all be regarded as equivalent replacements or derivatives of the technical solution claimed to be protected by the embodiments of the present invention.

[0062] The voice interaction device 100 in the embodiment of the present invention is mainly used to: respond to a voice comfort request triggered by a target object, and collect interactive voice data and object image data of the target object;

[0063] The emotion data determines the emotion recognition data of the target object according to the interactive speech features corresponding to the interactive speech data and the image emotion data corresponding to the object image data;

[0064] A target soothing operation is performed according to the emotion recognition data and an emotion soothing strategy corresponding to the emotion recognition data to obtain an emotion soothing result.

[0065] The voice interaction device 100 in the embodiment of the present invention can be an independent voice interaction device, such as a smart terminal such as a mobile phone, a tablet computer, a network device, a server, and a smart computer, or it can be a voice interaction network or a voice interaction cluster composed of multiple voice interaction devices.

[0066] The embodiments of the present application provide a voice interaction method, apparatus, device, and computer-readable storage medium, which are described in detail below.

[0067] It can be understood by those skilled in the art that Figure 1 The application environment shown in is only one of the application scenarios related to the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 1 More or fewer voice interaction devices, or voice interaction network connection relationships shown, such as Figure 1 Only one voice interaction device is shown. It can be understood that the scenario of the voice interaction method can also include one or more voice interaction devices, which are not specifically limited here. The voice interaction device 100 can also include a memory for storing interaction data and other data.

[0068] It should be noted that Figure 1 The scenario diagram of the voice interaction method shown is only an example. The scenario of the voice interaction method described in the embodiment of the present invention is to more clearly illustrate the technical solution of the embodiment of the present invention, and does not constitute a limitation on the technical solution provided by the embodiment of the present invention.

[0069] Based on the scenario of the above-mentioned voice interaction method, various embodiments of the voice interaction method disclosed in the present invention are proposed.

[0070] like Figure 2 As shown, Figure 2 This is a flow chart of an embodiment of a voice interaction method in an embodiment of the present application. The voice interaction method includes the following steps 201 to 203:

[0071] 201. In response to a voice soothing request triggered by a target object, collect interactive voice data and object image data of the target object;

[0072] The voice interaction method in this embodiment is applied to voice interaction devices, and the type and number of voice interaction devices are not specifically limited. That is, the voice interaction device can be one or more intelligent terminals including intelligent voice assistants that can perform voice interaction with users. In a specific embodiment, the voice interaction device is a smart TV. Optionally, in other embodiments, the voice interaction device can be a smart device such as a mobile phone, a smart computer, and a tablet computer.

[0073] Among them, the intelligent voice assistant can be a voice assistant application built based on technologies such as artificial intelligence (such as a generative large model), which can generate corresponding interactive content based on the interactive voice data of the target object, and output the interactive content using a voice output module or a display output module. That is, the voice interaction device can communicate with the user through the intelligent voice assistant, and perform corresponding interactive operations through the user's voice dialogue. For example, the user can wake up the intelligent voice assistant and ask questions to the intelligent voice assistant to obtain the answer (including text, recommended information, and picture information, etc.) generated by the intelligent voice assistant. Optionally, the user can also wake up the intelligent voice assistant, send an interactive instruction for the voice interaction device to the intelligent voice assistant, and drive the intelligent voice assistant to execute the interactive instruction. Optionally, the user can also perform other unmentioned interactions with the intelligent voice assistant, which is not specifically limited in this embodiment.

[0074] Specifically, during operation, the voice interaction device can respond to a voice soothing request triggered by a target object, wherein the voice soothing request is to drive the voice interaction device to detect the target object in the voice interaction scene triggered by the target object to identify the target object's emotion recognition data, and determine the target object's emotional state through the emotion recognition data, and perform an operation event of emotional soothing when the target object is in a depressed or angry emotional state. Among them, the triggering method of the voice soothing request is not specifically limited here, that is, the voice soothing request can be actively triggered by the user, for example, the user can click the voice soothing button in the intelligent voice assistant setting page corresponding to the voice interaction device, and actively trigger the voice soothing request. Optionally, in other embodiments, the voice soothing request can also be automatically triggered by the voice interaction device, for example, the voice interaction device is set by an emotional soothing process, and when it is detected that the user starts the intelligent voice assistant, the voice soothing request is automatically triggered. Among them, the target object is a user who performs voice interaction operations with the intelligent voice assistant in the voice interaction device.

[0075] Specifically, after the voice interaction device obtains the voice soothing request, when the target object uses the intelligent voice assistant to perform voice interaction operations, the target object's interactive voice data and the target object's object image data are collected. The interactive voice data is the voice data when the target object performs voice interaction with the intelligent voice assistant. The object image data is the face image data when the target object performs voice interaction with the intelligent voice assistant.

[0076] Specifically, the voice interaction device is provided with a voice acquisition module, an image acquisition module and a computer vision module. When the target object uses the intelligent voice assistant for voice interaction, the voice interaction device uses the voice acquisition module to collect the real-time interactive voice data of the target object. In addition, when the target object uses the intelligent voice assistant for voice interaction, the voice interaction device also uses the image acquisition module and the computer vision module to collect the object image data of the target object.

[0077] 202. The emotion data determines the emotion recognition data of the target object according to the interactive speech features corresponding to the interactive speech data and the image emotion data corresponding to the object image data;

[0078] Specifically, after obtaining the interactive voice data and object image data corresponding to the target object, the voice interaction device also uses the interactive voice data and interactive voice data to analyze the emotional state of the target object when using the intelligent voice assistant for voice interaction, and obtains emotion recognition data. That is, the voice interaction device determines the emotion recognition data of the target object based on the voice emotion data corresponding to the interactive voice data and the image emotion data corresponding to the object image data.

[0079] Specifically, after acquiring the interactive voice data and the object image data, the voice interaction device performs emotion classification on the interactive voice features corresponding to the interactive voice data, thereby determining the voice emotion data corresponding to the interactive voice features. That is, the voice interaction device is pre-set with a voice emotion recognition model capable of performing emotion classification on the interactive voice data, and after acquiring the interactive voice data, the interactive voice data is input into the voice emotion recognition model for emotion classification, thereby obtaining the voice emotion data corresponding to the interactive voice data.

[0080] Specifically, after the voice interaction device inputs the interactive voice data into the voice emotion recognition model, the voice emotion recognition model is used to extract voice features of the interactive voice data, that is, the voice encoding module in the voice emotion recognition model is used to encode the voice features of the interactive voice data, thereby obtaining the interactive voice features of the interactive voice data. The interactive voice features may be voice feature information such as a spectrogram, a Mel-frequency cepstral coefficient (MFCC), and a fundamental frequency (pitch) corresponding to the interactive voice data.

[0081] After acquiring the interactive speech feature, the speech interaction device also uses the initial emotion classification module in the speech emotion recognition model to perform initial emotion classification on the interactive speech feature, thereby determining the emotion category described in the interactive speech feature and obtaining the initial emotion feature of the interactive speech feature. The initial emotion feature can be divided into positive emotion features and negative emotion features.

[0082] Specifically, after using the initial emotion classification module to determine the initial emotion feature corresponding to the interactive voice feature, the voice interaction device also determines whether the initial emotion feature is a negative emotion feature, and then determines whether further emotion recognition and corresponding soothing operations are required for the initial emotion feature.

[0083] Optionally, if the voice interaction device determines that the initial emotional feature is a negative emotional feature, the voice interaction device performs secondary emotional classification on the initial emotional feature according to the voice recognition model, thereby obtaining voice emotion data corresponding to the interactive voice feature. The voice emotion data is a quantitative degree parameter that characterizes the negative emotional state of the target object when it is in a negative emotional state during voice interaction with the intelligent voice assistant. Optionally, in a specific embodiment, the voice emotion data includes a voice emotion level. The voice emotion level is a quantitative parameter that characterizes the negative emotional state and negative emotion level of the target object obtained through voice data.

[0084] Specifically, after determining that the initial emotional feature is a negative emotional feature, the voice interaction device also uses the voice convolution module and the recurrent neural network module in the voice recognition model to perform secondary emotional classification on the initial emotional feature to obtain the voice emotion data corresponding to the interactive voice feature. The voice convolution module can be a trained CNN model, and the recurrent neural network module can be an LSTM model.

[0085] Specifically, the voice interaction device can also perform facial expression recognition on the object image features corresponding to the object image data, thereby determining the user facial expression state of the target object when performing voice interaction with the intelligent voice assistant, so as to obtain image emotion data corresponding to the object image data, which characterizes the user emotional state of the target object. The image emotion data includes an image emotion level, which is a quantitative parameter characterizing the negative emotional state and degree of negative emotion of the target object obtained through the face image data.

[0086] Specifically, after acquiring the voice emotion data and the image emotion data, the voice interaction device also performs emotion matching in multiple dimensions based on the voice emotion data and the image emotion data, thereby determining the emotion recognition data of the target object, thereby improving the accuracy of emotion recognition of the target object.

[0087] Among them, the emotion recognition data is a quantitative parameter that characterizes the negative emotional state and degree of negative emotion of the target object. Optionally, in a specific embodiment, the emotion recognition data can be a specific emotion type and a target emotion level corresponding to the specific emotion type. For example, the emotion recognition data can be anger emotion level 1, anger emotion level 2...anger emotion level n, etc. In addition, the emotion recognition data can also be other emotion types and target emotion levels corresponding to the emotion types.

[0088] 203. Execute a target soothing operation according to the emotion recognition data and an emotion soothing strategy corresponding to the emotion recognition data to obtain an emotion soothing result.

[0089] Specifically, after obtaining the emotion recognition data of the target object when using the intelligent voice assistant for voice interaction, the voice interaction device also performs a target soothing operation according to the emotion recognition data and the emotion soothing strategy corresponding to the emotion recognition data to obtain an emotion soothing result.

[0090] Specifically, when the target object uses the intelligent voice assistant for voice interaction, the intelligent voice assistant fails to output the interaction content that satisfies the target object, causing the user to be angry or in other emotional states. After the voice interaction device recognizes the emotion recognition data, it analyzes the emotion recognition data, obtains the target emotion level in the emotion recognition data, generates an emotion soothing strategy corresponding to the target emotion level, outputs the corresponding emotion soothing content based on the emotion soothing strategy, and obtains the emotion soothing result. Among them, the emotion soothing level is an interaction strategy that drives the intelligent voice assistant to change the corresponding interaction content or perform a specified soothing interaction operation according to the target emotion level.

[0091] Specifically, after obtaining the emotion soothing strategy, the voice interaction device accesses the cloud database to obtain the emotion soothing content corresponding to the emotion soothing strategy and the interactive voice data in the cloud database. That is, the voice interaction device can obtain the emotion soothing content corresponding to the voice interaction scene that is the same as the emotion soothing strategy and the interactive voice data stored in the cloud database. Among them, the emotion soothing content can be to replace the correct interactive content corresponding to the interactive voice data, or to send an inquiry content in a questioning tone asking whether to search for specified content.

[0092] Specifically, after acquiring the emotion-soothing content, the voice interaction device can output the emotion-soothing content to the target object through a voice output module or a display output module, and identify the emotion update data of the target object, wherein the emotion update data is real-time emotion recognition data that represents the update of the target object after perceiving the emotion-soothing content.

[0093] Optionally, after acquiring the emotion update data, the voice interaction device also compares the emotion update data with the emotion recognition data to determine whether the emotional state of the target object has improved. If the emotional state of the target object has improved, the voice interaction device outputs an emotion soothing result indicating successful emotion soothing and records the emotion soothing content.

[0094] In this embodiment, the voice interaction device collects the interactive voice data and object image data of the target object by responding to the voice soothing request triggered by the target object; the emotional data determines the emotional recognition data of the target object according to the interactive voice features corresponding to the interactive voice data and the image emotional data corresponding to the object image data; the target soothing operation is performed according to the emotional recognition data and the emotional soothing strategy corresponding to the emotional recognition data to obtain the emotional soothing result. The emotional recognition data of the target object in the voice interaction process is obtained by acquiring the interactive voice data and object image data of the target object, and then the emotional recognition is performed in multiple dimensions according to the interactive voice data and the object image data, so as to determine the emotional recognition data of the target object in the voice interaction process, and the corresponding soothing operation is performed according to the emotional soothing strategy corresponding to the emotional mouse data, thereby reducing the probability of the user being angry during the voice interaction and improving the user's voice interaction experience and satisfaction.

[0095] like Figure 3 As shown, Figure 3 This is a flow chart of an embodiment of performing image emotion data recognition in the voice interaction method provided in the embodiment of the present application. Specifically, in this embodiment, the voice interaction method further includes steps 301 to 302:

[0096] 301. Perform facial feature extraction on the object image data to obtain object image features corresponding to the object image data;

[0097] 302. Perform expression recognition on the object image features according to the target convolution module, the stochastic gradient descent module and the target feature extractor to obtain image emotion data corresponding to the object image data.

[0098] Based on the above embodiments, in this embodiment, the voice interaction device can also perform expression recognition on the object image features corresponding to the object image data, so as to determine the user expression state of the target object when voice interacting with the intelligent voice assistant, so as to obtain image emotion data corresponding to the object image data, which characterizes the user emotional state of the target object.

[0099] Specifically, the voice interaction device pre-constructs an image recognition model for facial expression recognition of object image data, and generates a training data set by collecting facial image data of different negative emotions under multiple factors such as different age groups and different genders, and divides the facial image data and voice training data of the same scene in the training data set into the same level, and divides the facial image data according to different age groups and constructs corresponding training subsets, and then trains the image recognition model according to the training subset and the training data set, thereby obtaining a trained image recognition model.

[0100] Specifically, after acquiring the trained image recognition model, the voice interaction device also uses the image recognition model to extract facial features from the object image data to obtain the object image features corresponding to the object image data. That is, the voice interaction device uses the image recognition model to perform face detection on the object image data, and after detecting the face area image of the object image data, it extracts features from the face area image to obtain the object image features corresponding to the object image data.

[0101] Specifically, after acquiring the object image features, the voice interaction device also uses the image recognition model to perform expression recognition on the object image features, thereby acquiring image emotion data corresponding to the object image data.

[0102] Specifically, the image recognition model is also provided with a target convolution module, a random gradient descent module and a target feature extractor for performing expression recognition on image features. After acquiring the object image features, the image recognition model inputs the object image features into the target convolution module for convolution processing, and inputs the object image features after convolution processing into the random gradient descent module for optimization processing, and adds them into the target feature extractor for expression recognition, so as to obtain the image emotion data corresponding to the object image data. In a specific embodiment, the target convolution module is a CNN deep convolution model, the random gradient descent module can be an SGD optimization model, and the target feature extractor can be a VGG model.

[0103] Specifically, after acquiring the voice emotion data and the image emotion data, the voice interaction device also performs emotion matching in multiple dimensions based on the voice emotion data and the image emotion data, so as to determine the emotion recognition data of the target object, thereby improving the accuracy of emotion recognition of the target object.

[0104] In this embodiment, the voice interaction device extracts facial features from the object image data to obtain the object image features corresponding to the object image data; and performs expression recognition on the object image features according to the target convolution module, the random gradient descent module and the target feature extractor to obtain the image emotion data corresponding to the object image data. The target object's emotional state when using the intelligent voice assistant for voice interaction can be identified based on its facial expression features, providing data support for subsequent emotional comfort.

[0105] like Figure 4 As shown, Figure 4 This is a flow chart of an embodiment of performing emotion matching in the voice interaction method provided in the embodiment of the present application to determine the emotion recognition data of the target object. Specifically, in this embodiment, the voice interaction method includes steps 401 to 403:

[0106] 401. Obtain a voice emotion level corresponding to the voice emotion data, and obtain an image emotion level corresponding to the image emotion data;

[0107] 402. If the difference between the speech emotion level and the image emotion level is less than or equal to a preset first difference threshold, the speech emotion level or the image emotion level with a larger value among the speech emotion level and the image emotion level is used as the emotion recognition data of the target object;

[0108] 403. If the level difference between the voice emotion level and the image emotion level is greater than or equal to a preset second difference threshold, a speech recognition model is used to perform secondary recognition on the interactive speech data and the interactive speech data to obtain emotion recognition data of the target object.

[0109] Based on the above embodiments, in this embodiment, after acquiring the voice emotion data and the image emotion data, the voice interaction device also performs emotion matching in multiple dimensions based on the voice emotion data and the image emotion data, so as to determine the emotion recognition data of the target object, thereby improving the accuracy of emotion recognition of the target object.

[0110] Specifically, after acquiring the voice emotion data and the image emotion data, the voice interaction device acquires the voice emotion level corresponding to the voice emotion data and the image emotion level corresponding to the image emotion data, and determines the emotion recognition data of the target object by comparing the voice emotion level and the image emotion level. The voice emotion level is a quantitative parameter representing the negative emotional state and negative emotional degree of the target object acquired through the voice data. The image emotion level is a quantitative parameter representing the negative emotional state and negative emotional degree of the target object acquired through the face image data.

[0111] Specifically, the voice interaction device calculates the level difference between the voice emotion level and the image emotion level, and compares the level difference with a preset first difference threshold and a second difference threshold to determine whether the error between the voice emotion recognition and the image emotion recognition is too large, and then determines the emotion recognition data of the target object based on the comparison result.

[0112] Optionally, if the difference between the speech emotion level and the image emotion level is less than or equal to a preset first difference threshold, the voice interaction device determines that there is no error in the speech emotion recognition and the image emotion recognition, and then uses the speech emotion level or the image emotion level with the larger emotion level value between the speech emotion level and the image emotion level as the emotion recognition data of the target object. In a specific embodiment, the first difference threshold is 1.

[0113] That is, when the difference between the voice emotion level and the image emotion level is less than or equal to a preset first difference threshold, the voice interaction device compares the voice emotion level and the image emotion level, and if the voice emotion level is greater than the image emotion level, the voice emotion level is set as the emotion recognition data of the target object. If the voice emotion level is less than the image emotion level, the voice emotion level is set as the emotion recognition data of the target object.

[0114] Optionally, if the difference between the speech emotion level and the image emotion level is greater than or equal to a preset second difference threshold, the voice interaction device determines that there is a large error between the speech emotion recognition and the image emotion recognition, and the voice interaction device uses the speech recognition model to perform secondary recognition on the interactive speech data and the interactive speech data, thereby determining the emotion recognition data of the target object based on the recognition results, the speech emotion level and the image emotion level.

[0115] Specifically, the voice interaction device uses a voice recognition model to perform secondary recognition on the interactive voice data and the interactive voice data to obtain an updated emotion level, where the updated emotion level includes an updated voice emotion level and / or an updated image emotion level.

[0116] Specifically, after obtaining the updated emotion level, the voice interaction device calculates the voice emotion level, the image emotion level, and the average of the updated voice emotion level and / or the updated image emotion level in the updated emotion level to obtain the emotion recognition data of the target object.

[0117] In this embodiment, the voice interaction device obtains the voice emotion level corresponding to the voice emotion data and the image emotion level corresponding to the image emotion data; if the level difference between the voice emotion level and the image emotion level is less than or equal to the preset first difference threshold, the voice emotion level or the image emotion level with the larger level value between the voice emotion level and the image emotion level is used as the emotion recognition data of the target object; if the level difference between the voice emotion level and the image emotion level is greater than or equal to the preset second difference threshold, the speech recognition model is used to perform secondary recognition on the interactive voice data and the interactive voice data to obtain the emotion recognition data of the target object. The target emotion recognition result of the target object is determined according to the voice emotion recognition result and the image emotion recognition result, thereby reducing the error of the emotion recognition result.

[0118] In order to better implement the voice interaction method in the embodiment of the present application, based on the voice interaction method, a voice interaction device is also provided in the embodiment of the present application, such as Figure 5 As shown, Figure 5 The structure diagram of the voice interaction device provided in the embodiment of the present application is as follows. Specifically, the voice interaction device 500 includes:

[0119] The data collection module 501 is configured to respond to the voice soothing request triggered by the target object and collect the interactive voice data and object image data of the target object;

[0120] The emotion recognition module 502 is configured to determine the emotion recognition data of the target object according to the interactive voice features corresponding to the interactive voice data and the image emotion data corresponding to the object image data;

[0121] The voice interaction module 503 is configured to perform a target soothing operation according to the emotion recognition data and an emotion soothing strategy corresponding to the emotion recognition data to obtain an emotion soothing result.

[0122] In a possible implementation of this embodiment, the emotion data of the voice interaction device determines the emotion recognition data of the target object according to the interactive voice features corresponding to the interactive voice data and the image emotion data corresponding to the object image data, including:

[0123] Performing emotion classification on the interactive speech features corresponding to the interactive speech data to obtain speech emotion data corresponding to the interactive speech features;

[0124] Performing facial expression recognition on the object image features corresponding to the object image data to obtain image emotion data corresponding to the object image data;

[0125] Matching is performed based on the voice emotion data and the image emotion data to obtain emotion recognition data of the target object.

[0126] In a possible implementation of this embodiment, the voice interaction device performs emotion classification on the interactive voice features corresponding to the interactive voice data to obtain voice emotion data corresponding to the interactive voice features, including:

[0127] Extracting speech features from the interactive speech data to obtain interactive speech features of the interactive speech data;

[0128] Classifying the initial emotion of the interactive speech feature to obtain the initial emotion feature of the interactive speech feature;

[0129] If the initial emotional feature is a negative emotional feature, a secondary emotional classification is performed on the initial emotional feature according to a preset speech recognition model to obtain speech emotion data corresponding to the interactive speech feature.

[0130] In a possible implementation of this embodiment, the voice interaction device performs expression recognition on the object image features corresponding to the object image data to obtain the image emotion data corresponding to the object image data, including:

[0131] Extracting facial features from the object image data to obtain object image features corresponding to the object image data;

[0132] According to the target convolution module, the random gradient descent module and the target feature extractor, expression recognition is performed on the object image features to obtain image emotion data corresponding to the object image data.

[0133] In a possible implementation of this embodiment, the voice interaction device matches the voice emotion data and the image emotion data to obtain the emotion recognition data of the target object, including:

[0134] Acquire the voice emotion level corresponding to the voice emotion data, and acquire the image emotion level corresponding to the image emotion data;

[0135] If the difference between the speech emotion level and the image emotion level is less than or equal to a preset first difference threshold, the speech emotion level or the image emotion level with a larger value among the speech emotion level and the image emotion level is used as the emotion recognition data of the target object;

[0136] If the level difference between the voice emotion level and the image emotion level is greater than or equal to a preset second difference threshold, the speech recognition model is used to perform secondary recognition on the interactive speech data and the interactive speech data to obtain emotion recognition data of the target object.

[0137] In a possible implementation of this embodiment, the voice interaction device uses a voice recognition model to perform secondary recognition on the interactive voice data and the interactive voice data to obtain emotion recognition data of the target object, including:

[0138] Performing secondary recognition on the interactive voice data and the interactive voice data using a voice recognition model to obtain an updated emotion level, wherein the updated emotion level includes an updated voice emotion level and / or an updated image emotion level;

[0139] The speech emotion level, the image emotion level, the average value of the updated speech emotion level and / or the updated image emotion level are calculated to obtain the emotion recognition data of the target object.

[0140] In a possible implementation of this embodiment, the voice interaction device performs a target soothing operation according to the emotion recognition data and the emotion soothing strategy corresponding to the emotion recognition data to obtain an emotion soothing result, including:

[0141] Obtaining a target emotion level in the emotion recognition data, and generating an emotion soothing measurement strategy corresponding to the target emotion level;

[0142] Outputting corresponding emotion soothing content based on the emotion soothing strategy to obtain an emotion soothing result.

[0143] In a possible implementation of this embodiment, the voice interaction device outputs corresponding emotion soothing content based on the emotion soothing strategy to obtain an emotion soothing result, including:

[0144] Accessing a cloud database to obtain the emotion soothing strategy and the emotion soothing content corresponding to the interactive voice data;

[0145] Outputting the emotion soothing content to the target object and collecting emotion update data of the target object;

[0146] The emotion update data and the emotion recognition data are compared to obtain an emotion soothing result of the target object.

[0147] In this embodiment, the voice interaction device collects the interactive voice data and object image data of the target object by responding to the voice soothing request triggered by the target object; the emotional data determines the emotional recognition data of the target object according to the interactive voice features corresponding to the interactive voice data and the image emotional data corresponding to the object image data; the target soothing operation is performed according to the emotional recognition data and the emotional soothing strategy corresponding to the emotional recognition data to obtain the emotional soothing result. The emotional recognition data of the target object in the voice interaction process is obtained by acquiring the interactive voice data and object image data of the target object, and then the emotional recognition is performed in multiple dimensions according to the interactive voice data and the object image data, so as to determine the emotional recognition data of the target object in the voice interaction process, and the corresponding soothing operation is performed according to the emotional soothing strategy corresponding to the emotional mouse data, so as to reduce the probability of the user being angry during the voice interaction and improve the user's voice interaction experience and satisfaction.

[0148] The embodiment of the present invention also provides a voice interaction device, such as Figure 6 As shown, Figure 6 This is a schematic diagram of the structure of an embodiment of the voice interaction device provided in the embodiments of the present application.

[0149] The voice interaction device integrates any one of the voice interaction devices provided in the embodiments of the present invention, and the voice interaction device includes:

[0150] one or more processors;

[0151] Memory; and

[0152] One or more applications, wherein the one or more applications are stored in the memory and are configured so that the processor executes the steps of the voice interaction method described in any of the above-mentioned voice interaction method embodiments.

[0153] Specifically, the voice interaction device may include one or more processing core processors 601, one or more computer-readable storage media memories 602, a power supply 603, an input unit 604, and other components. Those skilled in the art will understand that Figure 6 The structure of the voice interaction device shown in the figure does not constitute a limitation on the voice interaction device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0154] The processor 601 is the control center of the voice interaction device. It uses various interfaces and lines to connect various parts of the entire voice interaction device. By running or executing software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, it executes various functions of the voice interaction device and processes data, thereby monitoring the voice interaction device as a whole. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601.

[0155] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the voice interaction device, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

[0156] The voice interaction device also includes a power supply 603 for supplying power to various components. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 603 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0157] The voice interaction device may also include an input unit 604, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0158] Although not shown, the voice interaction device may also include a display unit, etc., which will not be described in detail here. Specifically in this embodiment, the processor 601 in the voice interaction device will load the executable files corresponding to the processes of one or more applications into the memory 602 according to the following instructions, and the processor 601 will run the application stored in the memory 602 to implement various functions, as follows:

[0159] In response to a voice soothing request triggered by a target object, collecting interactive voice data and object image data of the target object;

[0160] The emotion data determines the emotion recognition data of the target object according to the interactive speech features corresponding to the interactive speech data and the image emotion data corresponding to the object image data;

[0161] A target soothing operation is performed according to the emotion recognition data and an emotion soothing strategy corresponding to the emotion recognition data to obtain an emotion soothing result.

[0162] To this end, an embodiment of the present invention provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any voice interaction method provided in an embodiment of the present invention. For example, the computer program can be loaded by a processor to execute the following steps:

[0163] In response to a voice soothing request triggered by a target object, collecting interactive voice data and object image data of the target object;

[0164] The emotion data determines the emotion recognition data of the target object according to the interactive speech features corresponding to the interactive speech data and the image emotion data corresponding to the object image data;

[0165] A target soothing operation is performed according to the emotion recognition data and an emotion soothing strategy corresponding to the emotion recognition data to obtain an emotion soothing result.

[0166] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.

[0167] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.

[0168] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0169] The above is a detailed introduction to a voice interaction method provided in an embodiment of the present application. Specific embodiments are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A voice interaction method, characterized in that: The voice interaction method comprises: In response to a voice soothing request triggered by a target object, collecting interactive voice data and object image data of the target object; The emotion data determines the emotion recognition data of the target object according to the interactive speech features corresponding to the interactive speech data and the image emotion data corresponding to the object image data; A target soothing operation is performed according to the emotion recognition data and an emotion soothing strategy corresponding to the emotion recognition data to obtain an emotion soothing result.

2. The voice interaction method according to claim 1, characterized in that: The emotion data determines the emotion recognition data of the target object according to the interactive speech features corresponding to the interactive speech data and the image emotion data corresponding to the object image data, including: Performing emotion classification on the interactive speech features corresponding to the interactive speech data to obtain speech emotion data corresponding to the interactive speech features; Performing facial expression recognition on the object image features corresponding to the object image data to obtain image emotion data corresponding to the object image data; Matching is performed based on the voice emotion data and the image emotion data to obtain emotion recognition data of the target object.

3. The voice interaction method according to claim 2, characterized in that: The performing emotion classification on the interactive speech features corresponding to the interactive speech data to obtain speech emotion data corresponding to the interactive speech features includes: Extracting speech features from the interactive speech data to obtain interactive speech features of the interactive speech data; Classifying the initial emotion of the interactive speech feature to obtain the initial emotion feature of the interactive speech feature; If the initial emotional feature is a negative emotional feature, a secondary emotional classification is performed on the initial emotional feature according to a preset speech recognition model to obtain speech emotion data corresponding to the interactive speech feature.

4. The voice interaction method according to claim 2, characterized in that: The performing expression recognition on the object image features corresponding to the object image data to obtain the image emotion data corresponding to the object image data includes: Extracting facial features from the object image data to obtain object image features corresponding to the object image data; According to the target convolution module, the random gradient descent module and the target feature extractor, expression recognition is performed on the object image features to obtain image emotion data corresponding to the object image data.

5. The voice interaction method according to claim 2, characterized in that: The step of matching the speech emotion data with the image emotion data to obtain emotion recognition data of the target object includes: Acquire the voice emotion level corresponding to the voice emotion data, and acquire the image emotion level corresponding to the image emotion data; If the difference between the speech emotion level and the image emotion level is less than or equal to a preset first difference threshold, the speech emotion level or the image emotion level with a larger value among the speech emotion level and the image emotion level is used as the emotion recognition data of the target object; If the level difference between the voice emotion level and the image emotion level is greater than or equal to a preset second difference threshold, the speech recognition model is used to perform secondary recognition on the interactive speech data and the interactive speech data to obtain emotion recognition data of the target object.

6. The voice interaction method according to claim 5, characterized in that: The using of the speech recognition model to perform secondary recognition on the interactive speech data and the interactive speech data to obtain emotion recognition data of the target object includes: Performing secondary recognition on the interactive voice data and the interactive voice data using a voice recognition model to obtain an updated emotion level, wherein the updated emotion level includes an updated voice emotion level and / or an updated image emotion level; The speech emotion level, the image emotion level, the average value of the updated speech emotion level and / or the updated image emotion level are calculated to obtain the emotion recognition data of the target object.

7. The voice interaction method according to any one of claims 1 to 6, characterized in that: The step of performing a target soothing operation according to the emotion recognition data and an emotion soothing strategy corresponding to the emotion recognition data to obtain an emotion soothing result includes: Obtaining a target emotion level in the emotion recognition data, and generating an emotion soothing measurement strategy corresponding to the target emotion level; Outputting corresponding emotion soothing content based on the emotion soothing strategy to obtain an emotion soothing result.

8. The voice interaction method according to claim 7, characterized in that: The step of outputting corresponding emotion soothing content based on the emotion soothing strategy to obtain an emotion soothing result includes: Accessing a cloud database to obtain the emotion soothing strategy and the emotion soothing content corresponding to the interactive voice data; Outputting the emotion soothing content to the target object and collecting emotion update data of the target object; The emotion update data and the emotion recognition data are compared to obtain an emotion soothing result of the target object.

9. A voice interaction device, characterized in that: The voice interaction device comprises: A data collection module, configured to respond to a voice soothing request triggered by a target object and collect interactive voice data and object image data of the target object; An emotion recognition module is configured to determine emotion recognition data of the target object according to the interactive voice features corresponding to the interactive voice data and the image emotion data corresponding to the object image data; The voice interaction module is configured to perform a target soothing operation according to the emotion recognition data and an emotion soothing strategy corresponding to the emotion recognition data to obtain an emotion soothing result.

10. A voice interaction device, characterized in that: The voice interaction device comprises: one or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the steps of the voice interaction method described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps of the voice interaction method according to any one of claims 1 to 8.

Images