Interactive dialogue method, device and equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides an interactive dialogue method and device, equipment and a storage medium, and the method comprises the steps: collecting a voice signal inputted by a user, and carrying out the recognition of the voice signal, and obtaining first text content corresponding to the voice signal; extracting voice emotion representation of the voice signal; extracting a text emotion representation of the first text content; fusing the speech emotion representation and the text emotion representation to obtain emotion information; according to the first text content, determining second text content which has a dialogue with the first text content; and outputting dialogue voice according to the emotion information and the second text content. According to the method, the voice emotion representation and the text emotion representation are fused, so that the real emotion state of the user is fully known, and the emotion information matched with the emotion state or the speaking style of the user is obtained. According to the emotional information and the second text content, the dialogue voice matched with the emotional state or the speaking style of the user is output, and the authenticity of the interactive dialogue is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an interactive dialogue method, device, equipment and storage medium. Background Art

[0002] With the rapid development of human-computer interaction in the field of artificial intelligence technology, intelligent dialogue systems and users can exchange information and provide feedback in real time. They can have two-way dialogues with users, understand users' questions and provide corresponding answers. In the medical field, patients can have interactive conversations with customer service robots equipped with intelligent dialogue systems in hospitals over the phone. For example, patients can ask customer service robots about hospital information such as the location of the clinic and the examination department. The customer service robots process the information consulted by users through pre-set programs and output voice responses.

[0003] In the related art, a fixed-speech style is generally used to synthesize a specified reply dialogue, which cannot generate an emotional dialogue that fits the actual emotion according to the user's current emotion. It is very different from the real dialogue scene, making the dialogue process appear boring and tedious. Summary of the invention

[0004] The main purpose of the present application is to provide an interactive dialogue method, apparatus, device and storage medium, which integrate voice emotion representation and text emotion representation, so as to fully understand the user's real emotional state and obtain emotional information that can match the user's emotional state or speaking style; based on the emotional information and the second text content, output a dialogue voice that matches the user's emotional state or speaking style, thereby improving the authenticity of the interactive dialogue.

[0005] In a first aspect, the present application provides an interactive dialogue method, comprising:

[0006] Collecting a voice signal input by a user, and recognizing the voice signal to obtain a first text content corresponding to the voice signal;

[0007] Extracting a speech emotion representation of the speech signal;

[0008] Extracting a text sentiment representation of the first text content;

[0009] The speech emotion representation is integrated with the text emotion representation to obtain emotion information;

[0010] Determining, based on the first text content, a second text content that dialogues with the first text content;

[0011] Outputting a dialogue voice according to the emotion information and the second text content.

[0012] In a second aspect, the present application further provides an interactive dialogue device, comprising:

[0013] A collection and recognition module, used to collect a voice signal input by a user, and recognize the voice signal to obtain a first text content corresponding to the voice signal;

[0014] A first extraction module, used to extract the speech emotion representation of the speech signal;

[0015] A second extraction module, used to extract text sentiment representation of the first text content;

[0016] A fusion module, used to fuse the speech emotion representation with the text emotion representation to obtain emotion information;

[0017] A text determination module, configured to determine, based on the first text content, a second text content that dialogues with the first text content;

[0018] A speech output module is used to output a dialogue speech according to the emotion information and the second text content.

[0019] In a third aspect, the present application further provides a computer device, the computer device comprising a memory and a processor;

[0020] The memory is used to store computer programs;

[0021] The processor is used to execute the computer program and implement the above-mentioned interactive dialogue method when executing the computer program.

[0022] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the interactive dialogue method as described above are implemented.

[0023] The present application provides an interactive dialogue method, device, equipment and storage medium, wherein the method includes: collecting a voice signal input by a user, and identifying the voice signal to obtain a first text content corresponding to the voice signal; extracting a voice emotion representation of the voice signal; extracting a text emotion representation of the first text content; fusing the voice emotion representation with the text emotion representation to obtain emotion information; determining a second text content that dialogues with the first text content according to the first text content; and outputting a dialogue voice according to the emotion information and the second text content. The present application first collects a voice signal input by a user and identifies the voice signal to obtain a first text content, then extracts a voice emotion representation of the voice signal and a text emotion representation of the first text content, and then fuses the voice emotion representation and the text emotion representation, so as to fully understand the user's true emotional state and obtain emotional information that can match the user's emotional state or speaking style. According to the first text content, determine the second text content that dialogues with the first text content, so that the second text content can reply to the first text content in a targeted manner, thereby improving the intelligence of the dialogue. Finally, according to the emotion information and the second text content, output a dialogue voice that matches the user's emotional state or speaking style, thereby improving the authenticity of the interactive dialogue. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A flowchart of an interactive dialogue method provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the connection between the server and the terminal device provided in the embodiment of the present application;

[0027] Figure 3 A schematic diagram of an interactive dialogue process provided by an embodiment of the present application;

[0028] Figure 4 A schematic block diagram of an interactive dialogue device provided in an embodiment of the present application;

[0029] Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0032] The embodiments of the present application provide an interactive dialogue method, apparatus, device and storage medium. The interactive dialogue method can be applied to a terminal device, which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer and the like. It can also be applied to a server, which can be a separate server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0033] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0034] See also Figure 1 , Figure 1 This is a flow chart of an interactive dialogue method provided in an embodiment of the present application. It should be noted that the interactive dialogue method provided in an embodiment of the present application can be used in a terminal device, and of course can also be used in a server.

[0035] like Figure 2 As shown, the interactive dialogue method is applied to a server, the server and the terminal device are connected in communication, and the dialogue voice obtained by the interactive dialogue method can be sent to the terminal device through the server. Of course, it is not limited to this and is not limited here.

[0036] In specific implementation, the terminal device includes but is not limited to: any one of a mobile phone, a tablet computer, a laptop computer, and a desktop computer; the server can be a single server or a server cluster, or a cloud server that provides cloud computing services.

[0037] like Figure 1 As shown, the interactive dialogue method includes steps S101 to S106.

[0038] Step S101: Collect a voice signal input by a user, and recognize the voice signal to obtain a first text content corresponding to the voice signal.

[0039] It should be noted that the embodiments of the present application can be applied to an intelligent dialogue system. An intelligent dialogue system is a computer program based on artificial intelligence technology that can simulate human language communication ability and realize natural dialogue with users. The embodiments of the present application can collect a voice signal input by a user, and recognize the voice signal input by the user through a voice recognition technology (such as automatic speech recognition) to convert the voice signal into a first text content corresponding to the voice signal.

[0040] Step S102: extracting speech emotion representation of the speech signal.

[0041] For example, after collecting a speech signal, the embodiment of the present application can extract a speech emotion representation of the speech signal.

[0042] It should be noted that speech emotion representation refers to reflecting the speaker's emotional state through specific features in the speech signal. By extracting the speech emotion representation of the speech signal, the user's real emotional state and speaking style can be understood. Specific features may include acoustic features such as prosodic features, sound quality features, and energy features. Among them, prosodic features refer to pitch, length, and speaking speed; for example, when you are happy, the pitch rises and the speaking speed increases; when you are sad, the pitch drops and the speaking speed slows down. Sound quality features refer to resonance peaks, spectrum tilts, etc., which can reflect the speaker's timbre and emotional state. Energy features are energy changes in speech signals. For example, the energy is higher when you are angry and lower when you are sad.

[0043] In an exemplary implementation, step S102 may include: step S201 and step S202.

[0044] Step S201: extract a first feature vector of a speech signal through a preset first feature extractor.

[0045] Step S202: input the first feature vector into a preset first emotion classification model, and obtain a speech emotion representation obtained by performing emotion classification on the first feature vector by the first emotion classification model.

[0046] For example, the embodiment of the present application can extract the first feature vector of the speech signal through a preset first feature extractor. Specifically, after collecting the speech signal, the speech signal can be preprocessed by denoising, filtering and normalization to improve the quality of the speech signal.

[0047] The feature extractor in the embodiment of the present application may include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), Transformer, etc. Among them, Transformer is a deep learning model for natural language processing. Unlike the traditional recurrent network RNN, Transformer uses an attention mechanism.

[0048] Exemplarily, the embodiments of the present application can extract the first eigenvector of the speech signal through methods such as Mel-frequency cepstral coefficients, Mel-frequency filter bank coefficients, linear prediction coefficients and deep learning.

[0049] The first feature vector is input into a preset first emotion classification model to perform emotion classification on the first feature vector based on the first emotion classification model, that is, the emotion category of the input first feature vector is predicted by the first emotion classification model to obtain the speech emotion representation corresponding to the first feature vector.

[0050] It is understandable that the embodiment of the present application can use the feature vector of the labeled speech signal as training data to train the first emotion classification model. During the training process, the first emotion classification model can learn how to predict the emotion category based on the input feature vector.

[0051] In the financial field, users can consult the customer service robot with an intelligent dialogue system in the insurance company over the phone about the purchase of insurance or related information about accidents. For example, the user says to the customer service robot over the phone, "I have just bought a gasoline car and need to buy car insurance. Can you recommend any affordable car insurance packages?" At this time, the voice signal input by the user during the consultation can be obtained first, and then the first feature vector of the user's voice signal (such as pitch, length, timbre, speaking speed, etc.) can be extracted. Then, the first feature vector is classified by the first emotion classification model to obtain the voice emotion representation, so as to judge the user's emotional state.

[0052] In an exemplary implementation, step S201 includes step S2011 and step S2012.

[0053] Step S2011: extract the acoustic features of the speech signal through a preset first feature extractor.

[0054] Step S2012: Based on the bottleneck network, perform dimensionality reduction processing on the acoustic features of the speech signal to obtain a first eigenvector of the speech signal.

[0055] Specifically, the embodiment of the present application can extract acoustic features of the speech signal through the first feature extractor, such as prosody features, sound quality features, energy features, etc. Different acoustic features include key information for emotion classification in the speech signal.

[0056] It should be noted that the bottleneck network is a structure used to reduce the number of model parameters, improve computational efficiency, and maintain model performance. The bottleneck network refers to inserting a "bottleneck" between convolutional layers. Specifically, the number of channels is first reduced through a 1x1 convolution kernel, and then feature extraction is performed through a 3x3 or other size convolution kernel. Finally, the number of channels is restored to the original number through a 1x1 convolution kernel, which can significantly reduce the number of parameters and the amount of computation.

[0057] Exemplarily, the embodiment of the present application is based on a bottleneck network and can perform dimensionality reduction processing on extracted acoustic features such as rhythmic features, sound quality features, and energy features, that is, by adjusting the number of channels by adjusting the size of the convolution kernel, thereby obtaining the first eigenvector of the speech signal, further reducing the number of parameters and the amount of calculation.

[0058] Step S103: extracting the text sentiment representation of the first text content.

[0059] In the embodiment of the present application, after the speech signal is recognized by speech recognition technology to obtain the first text content, the text emotion representation of the first text content can be extracted.

[0060] It should be noted that text sentiment representation refers to converting the emotional information contained in the text (for example, positive, negative, neutral, etc.) into computable and analyzable numerical values ​​or vectors, which helps the system understand and process the emotional information in the text to achieve sentiment classification.

[0061] In an exemplary implementation, step S1031 and step S1032.

[0062] Step S1031: extract a second feature vector of the first text content.

[0063] Step S1032: input the second feature vector into a preset second sentiment classification model, and obtain the text sentiment representation obtained by performing sentiment classification on the second feature vector by the second sentiment classification model.

[0064] Exemplarily, before extracting the second feature vector of the first text content, preprocessing operations such as text cleaning, word segmentation, stop word removal, and lemmatization can be performed on the first text content. Among them, text cleaning refers to removing irrelevant characters and noise data in the text, correcting spelling mistakes, and unifying the text format. Word segmentation refers to decomposing the first text content into words and vocabulary. For Chinese text, since there are no obvious delimiters between words, word segmentation is an important step in the preprocessing operation during the process of processing Chinese text. Specifically, the Jieba Chinese word segmentation tool can be used for word segmentation. Stop word removal refers to deleting common meaningless words in the first text content, such as stop words like "de", "he", "shi", etc., to reduce the dimension of the feature vector and noise. Lemmatization refers to restoring words to their basic forms (such as stems and roots) to reduce the impact of vocabulary variations on feature extraction.

[0065] For example, the embodiments of the present application can convert the preprocessed first text content into a feature vector based on the bag-of-words model, the Bert model, etc., and then perform processing operations such as normalization and dimensionality reduction on the feature vector to obtain the second feature vector. Among them, the Bert model is a bidirectional language representation model based on the Transformer architecture. It adopts the encoder structure of the Transformer and is a deep bidirectional model that can effectively capture the context information of the target word.

[0066] The embodiments of the present application can input the second feature vector into a preset second sentiment classification model, so that the second sentiment classification model performs sentiment classification based on the second feature vector and outputs a text sentiment representation.

[0067] In the medical field, a patient can communicate with a customer service robot with an intelligent dialogue system in the hospital through the hospital's hotline to consult the department information of the hospital. At this time, the voice signal input by the patient during the consultation can be first subjected to speech recognition to obtain the first text content. For example, the recognized first text content is "I want to have a physical examination at your hospital's headquarters or branch. May I ask which branch of your hospital is open this Saturday?" At this time, the second feature vector (such as a word vector) of the first text content can be extracted, and then the second sentiment classification model can perform sentiment classification on the second feature vector to obtain the text sentiment representation, so as to judge the user's emotional state.

[0068] In an exemplary embodiment, the second sentiment classification model in the embodiments of the present application can include a Bert model and a linear layer; when the second sentiment classification model performs sentiment classification on the second feature vector, the Bert model extracts the hidden features of the second feature vector, and the linear layer classifies the hidden features to obtain the text sentiment representation.

[0069] Specifically, the second sentiment classification model in the embodiment of the present application may include a Bert model and a linear layer. When the second feature vector is input into the second sentiment classification model composed of the Bert model and the linear layer, the Bert model can extract the latent features of the second feature vector, and the linear layer can classify the latent features, thereby obtaining text sentiment representations such as positive sentiment, negative sentiment, or neutral sentiment.

[0070] Exemplarily, the latent features of the second feature vector refer to deep, implicit features that are not directly reflected in the first text content, but need to be mined through algorithms or models. Hidden features play an important role in understanding the deep meaning of texts, capturing similarities and differences between texts, etc. For example, latent features may include semantic features, implicit relationship features, contextual features, etc. The embodiment of the present application can mine the latent features in the second feature vector after text vectorization through the Bert model, and classify each latent feature through a linear layer to obtain the corresponding text sentiment representation, thereby improving the accuracy of text sentiment classification.

[0071] In an exemplary embodiment, the method further includes step S301 and step S302.

[0072] Step S301: Acquire annotation data, where the annotation data includes a feature vector of a training text content and a text sentiment representation of the training text content, where the text sentiment representation includes at least one of the following: positive sentiment, negative sentiment, and neutral sentiment.

[0073] Step S302: fine-tune the second sentiment classification model according to the labeled data.

[0074] For example, an embodiment of the present application can first obtain the feature vector of the training text content after vectorization and the text sentiment representation of the training text content, and then fine-tune the second sentiment classification model composed of the Bert model and the linear layer according to the feature vector of the training text content and the text sentiment representation of different sentiment tags, so that the linear layer can output different text sentiment representations during text sentiment classification, such as positive sentiment, negative sentiment and neutral sentiment.

[0075] Step S104: Fusing the speech emotion representation with the text emotion representation to obtain emotion information.

[0076] The embodiment of the present application can integrate speech emotion representation with text emotion representation to obtain emotion information with higher emotion recognition accuracy.

[0077] Specifically, the embodiment of the present application can perform weighted summation or splicing fusion of speech emotion representation and text emotion representation. By integrating the emotion representation of two different modes, emotion information can be obtained based on the rhythmic features of speech (such as pitch, volume, speaking speed, etc.), sound quality features, and energy features, and emotion information can be obtained based on the semantic features of the text content (such as vocabulary, grammar, context, etc.), implicit relationship features, and context features. In this way, the emotional state of the user's current speech can be judged more accurately.

[0078] In an exemplary implementation, step S104 includes step S1041 and step S1042.

[0079] Step S1041: perform dimension adjustment processing on the speech emotion representation and / or the text emotion representation to obtain the speech emotion representation and the text emotion representation with the same dimension.

[0080] Step S1042: concatenate and fuse the speech emotion representation and text emotion representation of the same dimension to obtain emotion information.

[0081] Specifically, in the process of fusing the speech emotion representation and the text emotion representation, it is necessary to ensure that the speech and text data are synchronized in time to ensure the accuracy of the splicing and fusion of the speech emotion representation and the text emotion representation.

[0082] Exemplarily, the embodiments of the present application can perform dimensionality adjustment processing on speech emotion representation and text emotion representation. For example, only the dimension of speech emotion representation can be adjusted, or only the dimension of text emotion representation can be adjusted, or both the dimension of speech emotion representation and the dimension of text emotion representation can be adjusted at the same time, so that the dimension of speech emotion representation and the dimension of text emotion representation are the same, avoiding errors in fusion due to dimensional differences, thereby improving the fusion effect of emotion representations of different modalities.

[0083] In actual applications, the dimensions of the speech emotion representation and / or text emotion representation can be adjusted simply by filling zero values ​​and truncating longer feature vectors so that the dimensions of the two are the same.

[0084] In the medical or financial fields, after extracting the speech emotion representation of the user's input voice signal and the text emotion representation of the text content after voice recognition, the speech emotion representation and the text emotion representation can be spliced ​​and fused to simulate the situation of the user and the customer service robot in the medical scene or financial scene. By integrating the user's tone, speed, volume and specific words in the text content, the emotional information can be captured more comprehensively, so as to more accurately judge the emotional state of the user when inputting the voice signal.

[0085] Step S105: Determine, according to the first text content, a second text content that dialogues with the first text content.

[0086] In the embodiment of the present application, the first text content can be semantically understood through natural language processing technology, so as to determine the second text content that forms a dialogue with the first text content.

[0087] Step S106: output the dialogue voice according to the emotion information and the second text content.

[0088] After determining the emotional information corresponding to the voice signal input by the user and the second text content that is in dialogue with the first text content, the emotional information and the second text content are synthesized to output a dialogue voice. For example, when the emotional information is that the emotion is relatively low, a dialogue voice with a neutral emotional state can be output; when the emotional information is that the emotion is relatively excited, a dialogue voice with an excited emotional state can be output. Outputting the corresponding dialogue voice for the voice signal of different emotional states of the user can simulate a more natural dialogue scene, thereby improving the authenticity of the interactive dialogue.

[0089] like Figure 3 As shown, in response to the user's voice input operation, the user's voice signal is collected and voice recognition is performed to obtain text; then the voice emotion representation is extracted through the first emotion classification model; the text emotion representation of the text is extracted through the second emotion classification model; then, the voice emotion representation and the text emotion representation are fused to obtain emotion information; the dialogue text can be determined by performing language processing on the text obtained by voice recognition; finally, the dialogue text and the emotion information are subjected to speech synthesis, and a dialogue voice corresponding to the user's actual emotional state can be output.

[0090] The interactive dialogue method provided by the above embodiment includes: collecting the voice signal input by the user, and identifying the voice signal to obtain the first text content corresponding to the voice signal; extracting the voice emotion representation of the voice signal; extracting the text emotion representation of the first text content; fusing the voice emotion representation with the text emotion representation to obtain emotion information; determining the second text content that dialogues with the first text content according to the first text content; and outputting the dialogue voice according to the emotion information and the second text content. The present application first collects the voice signal input by the user and recognizes the voice signal to obtain the first text content, then extracts the voice emotion representation of the voice signal and the text emotion representation of the first text content, and then fuses the voice emotion representation with the text emotion representation, so as to fully understand the user's real emotional state, so as to obtain the emotion information that can match the user's emotional state or speaking style. According to the first text content, the second text content that dialogues with the first text content is determined, so that the second text content can reply to the first text content in a targeted manner, thereby improving the intelligence of the dialogue. Finally, according to the emotion information and the second text content, the dialogue voice that matches the user's emotional state or speaking style is output, thereby improving the authenticity of the interactive dialogue.

[0091] See also Figure 4 , Figure 4 A schematic block diagram of an interactive dialogue device provided in an embodiment of the present application. The interactive dialogue device can be configured in a server or a terminal device to execute the aforementioned interactive dialogue method.

[0092] like Figure 4 As shown, the interactive dialogue device includes: a collection and identification module 110, a first extraction module 120, a second extraction module 130, a fusion module 140, a text determination module 150 and a speech output module 160.

[0093] The collection and recognition module 110 is used to collect the voice signal input by the user, and recognize the voice signal to obtain the first text content corresponding to the voice signal.

[0094] The first extraction module 120 is used to extract the speech emotion representation of the speech signal.

[0095] The second extraction module 130 is used to extract the text sentiment representation of the first text content.

[0096] The fusion module 140 is used to fuse the speech emotion representation with the text emotion representation to obtain emotion information.

[0097] The text determination module 150 is used to determine, based on the first text content, a second text content that dialogues with the first text content.

[0098] The speech output module 160 is used to output the dialogue speech according to the emotion information and the second text content.

[0099] In an exemplary embodiment, the first extraction module 120 may include a first extraction submodule and a first classification submodule.

[0100] The first extraction submodule is used to extract a first feature vector of the speech signal through a preset first feature extractor.

[0101] The first classification submodule is used to input the first feature vector into a preset first emotion classification model, and obtain the speech emotion representation obtained by performing emotion classification on the first feature vector by the first emotion classification model.

[0102] In an exemplary embodiment, the first extraction submodule may include a speech extraction submodule and a dimensionality reduction processing submodule:

[0103] The speech extraction submodule is used to extract the acoustic features of the speech signal through a preset first feature extractor.

[0104] The dimensionality reduction processing submodule is used to perform dimensionality reduction processing on the acoustic features of the speech signal based on the bottleneck network to obtain the first eigenvector of the speech signal.

[0105] In an exemplary embodiment, the second extraction module may include a second extraction submodule and a second classification submodule.

[0106] The second extraction submodule is used to extract a second feature vector of the first text content.

[0107] The second classification submodule is used to input the second feature vector into a preset second sentiment classification model, and obtain the text sentiment representation obtained by performing sentiment classification on the second feature vector by the second sentiment classification model.

[0108] In an exemplary embodiment, the device may further include: an acquisition module and a fine-tuning module.

[0109] The acquisition module is used to acquire the annotated data, the annotated data includes the feature vector of the training text content and the text sentiment representation of the training text content, the text sentiment representation includes at least one of the following: positive sentiment, negative sentiment and neutral sentiment.

[0110] The fine-tuning module is used to fine-tune the second sentiment classification model according to the labeled data.

[0111] In an exemplary embodiment, the second sentiment classification model may include a Bert model and a linear layer; when the second sentiment classification model performs sentiment classification on the second feature vector, the Bert model extracts latent features of the second feature vector, and the linear layer classifies the latent features to obtain text sentiment representation.

[0112] In an exemplary embodiment, the fusion module may include a dimension adjustment submodule and a splicing and fusion submodule.

[0113] The dimension adjustment submodule is used to perform dimension adjustment processing on the speech emotion representation and / or the text emotion representation to obtain the speech emotion representation and the text emotion representation with the same dimension.

[0114] The concatenation and fusion submodule is used to concatenate and fuse the speech emotion representation and text emotion representation of the same dimension to obtain emotion information.

[0115] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0116] The method of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0117] Exemplarily, the above method and apparatus may be implemented in the form of a computer program, which may be run on a computer device.

[0118] See also Figure 5 , Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a server or a terminal device.

[0119] like Figure 5 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and an internal memory.

[0120] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute the steps of any interactive dialogue method.

[0121] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0122] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute the steps of any interactive dialogue method.

[0123] The network interface is used for network communication, such as sending assigned tasks.

[0124] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0125] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0126] In one embodiment, the processor is used to execute a computer program and implement the following steps when executing the computer program:

[0127] Collecting a voice signal input by a user, and recognizing the voice signal to obtain a first text content corresponding to the voice signal;

[0128] Extract speech emotion representation of speech signals;

[0129] Extracting a text sentiment representation of the first text content;

[0130] The speech emotion representation is integrated with the text emotion representation to obtain the emotion information;

[0131] Determine, based on the first text content, a second text content that dialogues with the first text content;

[0132] Output the conversational speech according to the emotional information and the second text content.

[0133] In an exemplary embodiment, extracting a speech emotion representation of a speech signal includes:

[0134] Extracting a first feature vector of the speech signal by a preset first feature extractor;

[0135] The first feature vector is input into a preset first emotion classification model, and a speech emotion representation obtained by performing emotion classification on the first feature vector by the first emotion classification model is obtained.

[0136] In an exemplary embodiment, extracting a first feature vector of a speech signal by a preset first feature extractor includes:

[0137] Extracting acoustic features of the speech signal by a preset first feature extractor;

[0138] Based on the bottleneck network, the acoustic features of the speech signal are reduced in dimension to obtain the first eigenvector of the speech signal.

[0139] In an exemplary embodiment, extracting a text sentiment representation of the first text content includes:

[0140] Extracting a second feature vector of the first text content;

[0141] The second feature vector is input into a preset second sentiment classification model, and a text sentiment representation obtained by performing sentiment classification on the second feature vector by the second sentiment classification model is obtained.

[0142] In an exemplary embodiment, the method further comprises:

[0143] Acquire annotated data, the annotated data including a feature vector of a training text content and a text sentiment representation of the training text content, the text sentiment representation including at least one of the following: positive sentiment, negative sentiment, and neutral sentiment;

[0144] Fine-tune the second sentiment classification model based on the labeled data.

[0145] In an exemplary embodiment, the second sentiment classification model includes a Bert model and a linear layer; when the second sentiment classification model performs sentiment classification on the second feature vector, the Bert model extracts latent features of the second feature vector, and the linear layer classifies the latent features to obtain text sentiment representation.

[0146] In an exemplary embodiment, the speech emotion representation and the text emotion representation are integrated to obtain emotion information, including:

[0147] Performing dimension adjustment processing on the speech emotion representation and / or the text emotion representation to obtain the speech emotion representation and the text emotion representation with the same dimension;

[0148] The speech emotion representation and text emotion representation of the same dimension are concatenated and fused to obtain emotion information.

[0149] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above description of the interactive dialogue can refer to the corresponding process in the embodiment of the aforementioned interactive dialogue method, and will not be repeated here.

[0150] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The method implemented when the computer program is executed by a processor can refer to the various embodiments of the interactive dialogue method of the present application.

[0151] The computer-readable storage medium may be an internal storage unit of the computer device of the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device.

[0152] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0153] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0154] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An interactive dialogue method, characterized in that: include: Collecting a voice signal input by a user, and recognizing the voice signal to obtain a first text content corresponding to the voice signal; Extracting a speech emotion representation of the speech signal; Extracting a text sentiment representation of the first text content; The speech emotion representation is integrated with the text emotion representation to obtain emotion information; Determining, based on the first text content, a second text content that dialogues with the first text content; Outputting a dialogue voice according to the emotion information and the second text content.

2. The interactive dialogue method according to claim 1, characterized in that: The extracting the speech emotion representation of the speech signal comprises: Extracting a first feature vector of the speech signal by a preset first feature extractor; The first feature vector is input into a preset first emotion classification model, and a speech emotion representation obtained by performing emotion classification on the first feature vector by the first emotion classification model is obtained.

3. The interactive dialogue method according to claim 2, characterized in that: The extracting a first feature vector of the speech signal by a preset first feature extractor includes: Extracting the acoustic features of the speech signal by a preset first feature extractor; Based on the bottleneck network, the acoustic features of the speech signal are subjected to dimensionality reduction processing to obtain a first feature vector of the speech signal.

4. The interactive dialogue method according to claim 1, characterized in that: The step of extracting the text sentiment representation of the first text content includes: Extracting a second feature vector of the first text content; The second feature vector is input into a preset second sentiment classification model, and a text sentiment representation obtained by performing sentiment classification on the second feature vector by the second sentiment classification model is obtained.

5. The interactive dialogue method according to claim 4, characterized in that: The method further comprises: Acquire annotated data, wherein the annotated data includes a feature vector of a training text content and a text sentiment representation of the training text content, wherein the text sentiment representation includes at least one of the following: positive sentiment, negative sentiment, and neutral sentiment; The second sentiment classification model is fine-tuned according to the labeled data.

6. The interactive dialogue method according to claim 4, characterized in that: The second sentiment classification model includes a Bert model and a linear layer; when the second sentiment classification model performs sentiment classification on the second feature vector, the Bert model extracts latent features of the second feature vector, and the linear layer classifies the latent features to obtain text sentiment representation.

7. The interactive dialogue method according to claim 1, characterized in that: The step of fusing the speech emotion representation with the text emotion representation to obtain emotion information includes: Performing dimension adjustment processing on the speech emotion representation and / or the text emotion representation to obtain the speech emotion representation and the text emotion representation with the same dimension; The speech emotion representation and the text emotion representation of the same dimension are concatenated and fused to obtain emotion information.

8. An interactive dialogue device, characterized in that: include: A collection and recognition module, used to collect a voice signal input by a user, and recognize the voice signal to obtain a first text content corresponding to the voice signal; A first extraction module, used to extract the speech emotion representation of the speech signal; A second extraction module, used to extract text sentiment representation of the first text content; A fusion module, used to fuse the speech emotion representation with the text emotion representation to obtain emotion information; A text determination module, configured to determine, based on the first text content, a second text content that dialogues with the first text content; A speech output module is used to output a dialogue speech according to the emotion information and the second text content.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the interactive dialogue method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the interactive dialogue method according to any one of claims 1 to 7 are implemented.

Images