Voice output method, device and equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of voice processing, is applied to an intelligent customer service voice output scene, and relates to a voice output method, device and equipment and a storage medium thereof. Performing preprocessing to obtain a preprocessed voice segment; inputting the preprocessed voice segment into a multi-modal information fusion model, and predicting an output emotion type of the target voice segment; and controlling the current pronunciation object to output the emotion type output target voice segment. The voice output method is applied to an intelligent voice customer service response scene, so that the intelligent voice customer service fully considers context information and obtains emotion information from audio data and text data, the output voice fragment emotion is richer and more fits the actual voice consultation question and answer scene, and meanwhile, the voice output efficiency is improved. In the intelligent voice customer service dialogue, an emotion type is introduced, the situation that the output intelligent voice is too mechanized is avoided, and the dialogue experience of customers is improved.

Description

Technical Field

[0001] The present application relates to the field of voice processing technology, and is applied to intelligent customer service voice output scenarios, and in particular to a voice output method, device, equipment and storage medium thereof. Background Art

[0002] With the rapid development of the Internet, traditional voice output is increasingly tending towards intelligent voice output, that is, using intelligent voice customer service to replace manual customer service to communicate with customers.

[0003] At present, intelligent voice customer service is mainly used in simple business consulting fields, such as financial business consulting fields, and most of the intelligent voice customer service refers to the script text for mechanical answers, and does not introduce emotional types into the intelligent voice customer service question and answer scenario; even if there is a way to introduce emotional types, it only uses voice emotion recognition technology to analyze and identify the customer's emotional state, so as to better understand the user's needs and situation. During the interaction process, the emotional state is not introduced into the answer of the intelligent voice customer service. Mechanical answers can easily cause user disgust and bring customers a very poor service experience, and cannot create a real voice consultation question and answer scenario. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a voice output method, device, equipment and storage medium thereof to solve the problem that in the existing intelligent voice customer service response scenario, the emotional state is not introduced into the answer of the intelligent voice customer service during the interaction process, and the mechanical answer is likely to cause user disgust, bring a very poor service experience to the customer, and fail to create a real voice consultation question-and-answer scenario.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a voice output method, which adopts the following technical solution:

[0006] A speech output method comprises the following steps:

[0007] Acquire a first voice segment, a second voice segment, and a third voice segment, wherein the first voice segment is at least one sentence previously output by the current pronunciation object, the second voice segment is at least one sentence of the dialogue pronunciation object, and the third voice segment is a voice segment to be output by the current pronunciation object, and the first voice segment, the second voice segment, and the third voice segment are continuous in time relationship;

[0008] Preprocessing the first speech segment, the second speech segment, and the third speech segment respectively to obtain the preprocessed first speech segment, the second speech segment, and the third speech segment;

[0009] Inputting the preprocessed first speech segment, the second speech segment and the third speech segment into a preset multimodal information fusion model, and predicting the output emotion type of the third speech segment according to the multimodal information fusion model;

[0010] The current pronunciation object is controlled to output the third voice segment with the output emotion type, wherein the current pronunciation object includes an intelligent voice customer service.

[0011] Furthermore, the step of preprocessing the first voice segment, the second voice segment and the third voice segment respectively to obtain the preprocessed first voice segment, the second voice segment and the third voice segment specifically includes:

[0012] The first speech segment, the second speech segment and the third speech segment before preprocessing are used as the current speech segments to be processed in sequence;

[0013] Performing audio and text separation processing on the current voice segment to be processed to obtain separated pairs of audio data and text data;

[0014] Determine whether the audio data of the current voice segment to be processed exceeds a preset time length;

[0015] If the audio data of the current voice segment to be processed does not exceed the preset time length, a silent segment is introduced to fill the audio data to obtain target audio data consistent with the preset time length;

[0016] If the audio data of the current voice segment to be processed exceeds the preset time length, the audio data is intercepted and processed to obtain target audio data consistent with the preset time length;

[0017] The target audio data and the text data are used as the preprocessed speech segment of the current speech segment to be processed.

[0018] Furthermore, the step of performing audio and text separation processing on the current voice segment to be processed to obtain separated paired audio data and text data specifically includes:

[0019] Inputting the current speech segment to be processed into a preset ASR speech recognition model to obtain audio data and text data output by the ASR speech recognition model;

[0020] Marking the audio data and text data with the distinguishing identification information of the current voice segment to be processed, and obtaining the marked paired audio data and text data;

[0021] The step of introducing a silent segment to fill the audio data to obtain target audio data consistent with the preset time length specifically includes:

[0022] The silent segment is spliced ​​with the endpoints of the audio data to obtain new audio data in the format of "silent segment + audio data" or "audio data + silent segment" as the target audio data, wherein the time length of the new audio data is consistent with the preset time length, and the time length of the silent segment is the difference between the time length of the audio data and the preset time length.

[0023] Furthermore, the multimodal information fusion model includes an audio feature encoding component, a text feature encoding component, a feature vector splicing and fusion component, and a speech output component. Before executing the step of inputting the preprocessed first speech segment, the second speech segment, and the third speech segment into a preset multimodal information fusion model, and predicting the output emotion type of the third speech segment according to the multimodal information fusion model, the method further includes:

[0024] Obtain speech datasets of different emotion types;

[0025] Preprocessing each speech segment in the speech data set to obtain a speech data set after preprocessing;

[0026] Inputting the preprocessed speech data set into the multimodal information fusion model;

[0027] By means of the audio feature encoding component, the audio feature of each speech segment in the speech data set is encoded to obtain audio feature encoding information corresponding to each speech segment;

[0028] By means of the text feature encoding component, the text feature of each speech segment in the speech data set is encoded to obtain text feature encoding information corresponding to each speech segment;

[0029] The feature vector splicing and fusion component is used to splice and fuse the audio feature coding information and the text feature coding information of the same speech segment to obtain a fused final feature vector;

[0030] The final feature vectors corresponding to different speech segments of the same emotion type are clustered and sorted to obtain the final feature vector sets corresponding to different emotion types;

[0031] According to the final feature vector sets corresponding to different emotion types, the feature value ranges corresponding to different emotion types are determined;

[0032] According to the feature value ranges corresponding to different emotion types, an emotion type classifier is set in the fully connected layer corresponding to the speech output component, wherein setting the emotion type classifier means setting different speech output nodes for speech segments of different emotion types according to the feature value ranges in the fully connected layer corresponding to the speech output component.

[0033] Furthermore, the step of inputting the preprocessed first voice segment, the second voice segment and the third voice segment into a preset multimodal information fusion model, and predicting the output emotion type of the third voice segment according to the multimodal information fusion model specifically includes:

[0034] By means of the audio feature encoding component, the audio features of the preprocessed first voice segment, the second voice segment and the third voice segment are respectively encoded to obtain audio feature encoding information corresponding to each voice segment;

[0035] By means of the text feature encoding component, the text features of the preprocessed first voice segment, the second voice segment and the third voice segment are respectively encoded to obtain text feature encoding information corresponding to each voice segment;

[0036] Using the feature vector splicing and fusion component, splicing and fusing the audio feature coding information and the text feature coding information of the third voice segment to obtain a fused final feature vector;

[0037] Performing eigenvalue calculation on the final eigenvector after fusion to obtain a target eigenvalue, and identifying an eigenvalue range in which the target eigenvalue is located;

[0038] The output emotion type corresponding to the third speech segment is determined according to the feature value range of the target feature value.

[0039] Furthermore, before executing the step of using the feature vector splicing and fusion component to splice and fuse the audio feature coding information and the text feature coding information of the third voice segment to obtain the fused final feature vector, the method further includes:

[0040] Introducing a contextual attention mechanism into the audio feature encoding component and the text feature encoding component;

[0041] By using the context attention mechanism, the context dependency relationship between the audio feature encoding information of the third voice segment and the audio feature encoding information of the first voice segment and the second voice segment is calculated to obtain the audio context dependency relationship; and

[0042] By using the context attention mechanism, the context dependency relationship between the text feature encoding information of the third voice segment and the text feature encoding information of the first voice segment and the second voice segment is calculated to obtain the text context dependency relationship;

[0043] adding the audio context dependency as audio implicit information to the audio feature coding information of the third voice segment to update the audio feature coding information of the third voice segment;

[0044] adding the text context dependency as text implicit information to the text feature coding information of the third voice segment to update the text feature coding information of the third voice segment;

[0045] The feature vector splicing and fusion component includes a feature vector splicing and fusion component based on a cross-modal attention mechanism, and the step of using the feature vector splicing and fusion component to splice and fuse the audio feature encoding information and the text feature encoding information of the third voice segment to obtain the fused final feature vector specifically includes:

[0046] The audio feature encoding information of the third voice segment is used as a query vector, and the text feature encoding information of the third voice segment is used as a key vector and a value vector, and the correlation between the audio feature encoding information and the text feature encoding information of the third voice segment is calculated according to the attention mechanism, and feature weights are assigned to the text feature encoding information of the third voice segment according to the correlation to obtain a first feature weight assignment matrix;

[0047] Taking the text feature encoding information of the third voice segment as the Query vector, taking the audio feature encoding information of the third voice segment as the Key vector and the Value vector, and calculating the correlation between the text feature encoding information and the audio feature encoding information of the third voice segment according to the attention mechanism, and performing feature weight allocation on the audio feature encoding information of the third voice segment according to the correlation to obtain a second feature weight allocation matrix;

[0048] The first feature weight allocation matrix and the second feature weight allocation matrix are concatenated and fused using a matrix fusion method to obtain the final feature vector.

[0049] Furthermore, the step of controlling the current pronunciation object to output the third voice segment with the output emotion type specifically includes:

[0050] Filtering a target speech output node according to the output emotion type corresponding to the third speech segment;

[0051] The current pronunciation object is controlled to output the third voice segment through the target voice output node, wherein corresponding speaking tone, speaking pitch, sound intensity and sound speed are respectively set for different voice output nodes according to different emotion types.

[0052] In order to solve the above technical problems, the embodiment of the present application also provides a voice output device, which adopts the following technical solution:

[0053] A speech output device, comprising:

[0054] A voice segment acquisition module, used to acquire a first voice segment, a second voice segment and a third voice segment, wherein the first voice segment is at least one sentence previously output by the current pronunciation object, the second voice segment is at least one sentence of the dialogue pronunciation object, and the third voice segment is a voice segment to be output by the current pronunciation object, and the first voice segment, the second voice segment and the third voice segment are continuous in time relationship;

[0055] A speech segment preprocessing module, used to preprocess the first speech segment, the second speech segment and the third speech segment respectively to obtain the preprocessed first speech segment, the second speech segment and the third speech segment;

[0056] An output emotion prediction module is used to input the preprocessed first voice segment, the second voice segment and the third voice segment into a preset multimodal information fusion model, and predict the output emotion type of the third voice segment according to the multimodal information fusion model;

[0057] A voice segment output module is used to control the current pronunciation object to output the third voice segment with the output emotion type, wherein the current pronunciation object includes intelligent voice customer service.

[0058] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0059] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned voice output method when executing the computer-readable instructions.

[0060] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0061] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the voice output method as described above.

[0062] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0063] The speech output method described in the embodiment of the present application is to obtain a first speech segment, a second speech segment and a third speech segment, wherein the first speech segment, the second speech segment and the third speech segment are speech segments in the form of a question-and-answer dialogue; perform preprocessing to obtain the preprocessed first speech segment, the second speech segment and the third speech segment; input the preprocessed first speech segment, the second speech segment and the third speech segment into a multimodal information fusion model to predict the output emotion type of the third speech segment; and control the current pronunciation object to output the third speech segment with the output emotion type. Applying the speech output method of the present application to the intelligent voice customer service consultation and response scenario can make the response statement of the intelligent voice customer service fully consider the context information, obtain the sentence emotion information from the audio data and the text data, so that the speech segment output by the intelligent voice customer service is more emotionally rich and more in line with the actual voice consultation and question-and-answer scenario. At the same time, in the intelligent voice customer service dialogue, the emotion type is introduced to avoid the output intelligent voice being too mechanical, thereby improving the customer's dialogue experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0065] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0066] Figure 2 is a flow chart of an embodiment of a voice output method according to the present application;

[0067] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 202 is shown;

[0068] Figure 4 A flowchart of a specific embodiment of introducing output emotion type in the speech output method described in the present application is shown;

[0069] Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 203 is shown;

[0070] Figure 6 yes Figure 5 A flowchart of a specific embodiment of step 503 is shown;

[0071] Figure 7 is a structural schematic diagram of an embodiment of a speech output device according to the present application;

[0072] Figure 8 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0074] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0075] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0076] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0077] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0078] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0079] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0080] It should be noted that the voice output method provided in the embodiments of the present application is generally executed by a terminal device, and accordingly, the voice output device is generally provided in the terminal device.

[0081] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0082] Continue to refer Figure 2 , shows a flow chart of an embodiment of a voice output method according to the present application. The voice output method comprises the following steps:

[0083] Step 201: Acquire a first voice segment, a second voice segment, and a third voice segment.

[0084] Among them, the first voice segment is at least one sentence previously output by the current pronunciation object, the second voice segment is at least one sentence of the dialogue pronunciation object, and the third voice segment is the voice segment to be output by the current pronunciation object. The first voice segment, the second voice segment and the third voice segment are continuous in time relationship.

[0085] In this embodiment, the current pronunciation object includes intelligent voice customer service, the dialogue pronunciation object includes the consulting user who is having a dialogue with the intelligent voice customer service, the first voice segment is the last sentence output by the intelligent voice customer service, the second voice segment is the latest sentence asked by the consulting user based on the last sentence output by the intelligent voice customer service, and the third voice segment is the response sentence to be output by the intelligent voice customer service. Applying the voice output method described in this application to the intelligent voice customer service consultation and response scenario can make the response sentence of the intelligent voice customer service fully consider the context information, not only can the sentence emotion information be obtained from the audio data, but also can be obtained from the text data, so that the output third voice segment is more emotional and more in line with the actual voice consultation question and answer scenario. At the same time, in the intelligent voice customer service dialogue, the emotional type is introduced to avoid the output intelligent voice being too mechanical, and improve the customer's dialogue experience.

[0086] Step 202: pre-process the first voice segment, the second voice segment and the third voice segment respectively to obtain the pre-processed first voice segment, the second voice segment and the third voice segment.

[0087] Continue to refer Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 202 shown includes:

[0088] Step 301, taking the first voice segment, the second voice segment and the third voice segment before preprocessing in sequence as the current voice segment to be processed;

[0089] Step 302, performing audio and text separation processing on the current voice segment to be processed to obtain separated pairs of audio data and text data;

[0090] In this embodiment, the step of performing audio and text separation processing on the current voice segment to be processed to obtain paired audio data and text data after separation specifically includes: inputting the current voice segment to be processed into a preset ASR (Auto Speech Recognize) speech recognition model to obtain audio data and text data output by the ASR speech recognition model; marking the audio data and text data with distinguishing identification information of the current voice segment to be processed to obtain paired audio data and text data after marking.

[0091] Specifically, the first voice segment, the second voice segment and the third voice segment are taken as the current voice segments to be processed in turn, and the audio data and text data output by the ASR speech recognition model are marked with the corresponding audio data and text data according to the distinguishing identification information of the first voice segment, the second voice segment and the third voice segment, so as to facilitate the subsequent processing steps to distinguish the audio data and text data of the first voice segment, the second voice segment and the third voice segment.

[0092] Step 303, determining whether the audio data of the current voice segment to be processed exceeds a preset time length;

[0093] In this embodiment, the preset time length is, for example, 5 seconds, 8 seconds, etc.

[0094] Step 304: if the audio data of the current voice segment to be processed does not exceed the preset time length, a silent segment is introduced to fill the audio data to obtain target audio data consistent with the preset time length;

[0095] In this embodiment, the step of introducing a silent segment to complete the audio data to obtain target audio data consistent with the preset time length specifically includes: splicing the silent segment with the endpoints of the audio data to obtain new audio data in the format of "silent segment + audio data" or "audio data + silent segment" as the target audio data, wherein the time length of the new audio data is consistent with the preset time length, and the time length of the silent segment is the difference between the time length of the audio data and the preset time length.

[0096] Specifically, the target audio data is obtained by splicing the silent segment with the endpoints of the audio data, so that when the audio data does not exceed a preset time length, the integrity of the original audio data is guaranteed during the splicing process.

[0097] Step 305: if the audio data of the current voice segment to be processed exceeds the preset time length, the audio data is intercepted and processed to obtain target audio data consistent with the preset time length;

[0098] In this embodiment, the step of intercepting the audio data to obtain target audio data consistent with the preset time length specifically includes: framing the audio data according to a preset frame value interval to obtain audio data after framing; marking key frames and ordinary frames according to audio fluctuations between adjacent frames of audio data; and obtaining target audio data consistent with the preset time length by deleting the audio data corresponding to the ordinary frames.

[0099] Specifically, the audio data is divided into frames and key frames and ordinary frames are marked according to the audio fluctuations between adjacent frames of audio data. The target audio data consistent with the preset time length is obtained by deleting the audio data corresponding to the ordinary frames, thereby ensuring that when the original audio data exceeds the preset time length, more key frame audio data is retained as much as possible when it is deleted to avoid distortion of the original audio data.

[0100] Step 306: Use the target audio data and the text data as the pre-processed speech segment of the current speech segment to be processed.

[0101] By preprocessing the voice segment in advance, it is ensured that the time lengths of all audio data subsequently input into the multimodal information fusion model are consistent.

[0102] In this embodiment, the preprocessing further includes: performing text error correction processing on the text data, and performing coding length consistency processing on the text data according to the coding length of the target audio data.

[0103] Specifically, by performing text error correction processing on the text data and performing encoding length consistency processing on the text data according to the encoding length of the target audio data, it is easier to perform cross-modal interactive processing on the audio data and text data during subsequent voice output.

[0104] Step 203: input the preprocessed first voice segment, second voice segment and third voice segment into a preset multimodal information fusion model, and predict the output emotion type of the third voice segment according to the multimodal information fusion model.

[0105] In this embodiment, the multimodal information fusion model includes an audio feature encoding component, a text feature encoding component, a feature vector splicing and fusion component, and a speech output component.

[0106] Specifically, the audio feature encoding component is composed of a one-dimensional CNN convolutional neural network. The number of convolutional layers of the CNN convolutional neural network is 3. The convolution kernel sizes of the 3 convolutional layers are 2, 2, and 4, respectively. The convolution steps are 2, 2, and 2, respectively. The number of convolution channels is 64, 128, and 256, respectively. Finally, an audio feature matrix sequence of [216, 256] is output as audio feature encoding information.

[0107] The text feature encoding component includes a pre-trained BERT encoding component, and the step of obtaining text feature encoding information corresponding to each voice segment specifically includes: performing text encoding processing on the text data corresponding to the current voice segment through the pre-trained BERT encoding component to obtain a text embedding sequence corresponding to the current voice segment; performing self-attention weighted processing on the text embedding sequence corresponding to the current voice segment through a self-attention mechanism to obtain a coding sequence after self-attention weighted processing as the text feature encoding information of the current voice segment.

[0108] Continue to refer Figure 4 , Figure 4 A flowchart of a specific embodiment of introducing the output emotion type in the speech output method described in the present application is shown, including:

[0109] Step 401, obtaining speech data sets of different emotion types;

[0110] In this embodiment, the different emotion types are, for example, four different emotion types, namely, sadness, anger, neutrality and happiness.

[0111] Step 402, preprocessing each speech segment in the speech data set to obtain a speech data set after preprocessing;

[0112] Specifically, the preprocessing may be performed in the manner described in step 202 .

[0113] Step 403, inputting the preprocessed speech data set into the multimodal information fusion model;

[0114] In this embodiment, the multimodal information fusion model can be a self-constructed multimodal information fusion model or an audio-text information fusion model, for example: AudioPaLM audio-text information fusion model, wherein the AudioPaLM audio-text information fusion model is a bimodal information fusion model based on the Transformer architecture, which can fuse the audio and text information in the speech. Specifically, the multimodal information fusion model at least satisfies the requirement of being able to fuse the audio and text bimodal information.

[0115] Step 404: Encode the audio features of each speech segment in the speech data set through the audio feature encoding component to obtain audio feature encoding information corresponding to each speech segment;

[0116] Specifically, the audio data corresponding to each speech segment in the speech data set is input into the audio feature encoding component, and each audio data is convolved by the CNN convolutional neural network in the audio feature encoding component. After that, pooling and normalization are performed to obtain the audio feature matrix sequence corresponding to each audio data as audio feature encoding information.

[0117] Step 405, encoding the text features of each speech segment in the speech data set through the text feature encoding component to obtain text feature encoding information corresponding to each speech segment;

[0118] Specifically, the pre-trained BERT encoding component is used to perform text encoding processing on the text data corresponding to each speech segment in the speech data set to obtain a text embedding sequence corresponding to each speech segment; the self-attention mechanism is used to perform self-attention weighted processing on the text embedding sequence corresponding to each speech segment to obtain the encoding sequence after self-attention weighted processing as the text feature encoding information of each speech segment.

[0119] Step 406, using the feature vector concatenation and fusion component to concatenate and fuse the audio feature coding information and the text feature coding information of the same speech segment to obtain a fused final feature vector;

[0120] Specifically, the splicing and fusing of the audio feature coding information and the text feature coding information of the same voice segment to obtain the final fused feature vector can adopt a multimodal fusion method to obtain the final fused feature vector. The multimodal fusion method is specifically implemented as follows: the audio feature coding information of the same voice segment is used as the Query vector, and the text feature coding information is used as the Key vector and the Value vector, and according to the attention mechanism, the correlation between the audio feature coding information and the text feature coding information of the same voice segment is calculated, and the feature weights of the text feature coding information of the voice segment are assigned according to the correlation to obtain a first feature weight assignment matrix; the text feature coding information of the voice segment is used as the Query vector, and the audio feature coding information is used as the Key vector and the Value vector, and according to the attention mechanism, the correlation between the text feature coding information and the audio feature coding information of the voice segment is calculated, and the feature weights of the audio feature coding information of the voice segment are assigned according to the correlation to obtain a second feature weight assignment matrix; the first feature weight assignment matrix and the second feature weight assignment matrix are spliced ​​and fused by the matrix fusion method to obtain the final feature vector.

[0121] Specifically, the matrix fusion method may adopt a matrix value weighted average method, a matrix value weighted sum method or a matrix value weighted multiplication method.

[0122] Step 407, clustering and sorting the final feature vectors corresponding to different speech segments of the same emotion type to obtain final feature vector sets corresponding to different emotion types;

[0123] Step 408, determining the feature value ranges corresponding to the different emotion types according to the final feature vector sets corresponding to the different emotion types;

[0124] Step 409, according to the feature value ranges corresponding to different emotion types, an emotion type classifier is set in the fully connected layer corresponding to the speech output component, wherein setting the emotion type classifier means setting different speech output nodes for speech segments of different emotion types according to the feature value ranges in the fully connected layer corresponding to the speech output component.

[0125] By setting an emotion type classifier in advance according to the feature value ranges corresponding to different emotion types, in the fully connected layer corresponding to the voice output component, the subsequent intelligent voice customer service can select the appropriate voice output node for dialogue output according to the different emotion types.

[0126] Continue to refer Figure 5 , Figure 5 yes Figure 2 The flowchart of a specific embodiment of step 203 shown includes:

[0127] Step 501, encoding the audio features of the preprocessed first voice segment, the second voice segment, and the third voice segment respectively through the audio feature encoding component to obtain audio feature encoding information corresponding to each voice segment;

[0128] Step 502, encoding the text features of the preprocessed first voice segment, the second voice segment, and the third voice segment respectively through the text feature encoding component to obtain text feature encoding information corresponding to each voice segment;

[0129] Step 503, using the feature vector splicing and fusion component to splice and fuse the audio feature coding information and the text feature coding information of the third voice segment to obtain a fused final feature vector;

[0130] In this embodiment, before executing the step of using the feature vector splicing and fusion component to splice and fuse the audio feature coding information and text feature coding information of the third voice segment to obtain the final fused feature vector, the method also includes: introducing a contextual attention mechanism in the audio feature coding component and the text feature coding component; calculating the contextual dependency between the audio feature coding information of the third voice segment and the audio feature coding information of the first voice segment and the second voice segment through the contextual attention mechanism to obtain an audio context dependency; and calculating the contextual dependency between the text feature coding information of the third voice segment and the text feature coding information of the first voice segment and the second voice segment through the contextual attention mechanism to obtain a text context dependency; adding the audio context dependency as audio implicit information to the audio feature coding information of the third voice segment to update the audio feature coding information of the third voice segment; adding the text context dependency as text implicit information to the text feature coding information of the third voice segment to update the text feature coding information of the third voice segment.

[0131] Specifically, before splicing and fusing the audio feature coding information and text feature coding information of the third voice segment, a contextual attention mechanism is introduced into the audio feature coding component and the text feature coding component, so that when the audio feature coding information and text feature coding information of the third voice segment are fused, the feature coding information and context dependency of the first voice segment and the second voice segment are more combined, thereby ensuring that the third voice segment finally output by the intelligent voice customer service is more combined with the previous round of dialogue information.

[0132] Continue to refer Figure 6 , Figure 6 yes Figure 5 The flowchart of a specific embodiment of step 503 shown includes:

[0133] Step 601, using the audio feature encoding information of the third voice segment as a query vector, using the text feature encoding information of the third voice segment as a key vector and a value vector, and calculating the correlation between the audio feature encoding information and the text feature encoding information of the third voice segment according to an attention mechanism, and performing feature weight allocation on the text feature encoding information of the third voice segment according to the correlation to obtain a first feature weight allocation matrix;

[0134] Step 602: using the text feature encoding information of the third voice segment as a query vector, using the audio feature encoding information of the third voice segment as a key vector and a value vector, and calculating the correlation between the text feature encoding information and the audio feature encoding information of the third voice segment according to an attention mechanism, and performing feature weight allocation on the audio feature encoding information of the third voice segment according to the correlation to obtain a second feature weight allocation matrix;

[0135] Step 603: Concatenate and fuse the first feature weight allocation matrix and the second feature weight allocation matrix using a matrix fusion method to obtain the final feature vector.

[0136] It should be understood that the audio feature coding information in step 601 and step 602 is the audio feature coding information updated according to the audio context dependency, and the text feature coding information is also the text feature coding information updated according to the text context dependency.

[0137] Specifically, the audio feature encoding information of the third voice segment is used as the Query vector, the text feature encoding information of the third voice segment is used as the Key vector and the Value vector, and the text feature encoding information of the third voice segment is used as the Query vector and the audio feature encoding information of the third voice segment is used as the Key vector and the Value vector, and cross-modal information fusion is performed twice. Finally, the final fusion is performed to obtain the final feature vector, so that the fusion of audio information and text information is more complete.

[0138] Step 504, performing eigenvalue calculation on the final eigenvector after fusion to obtain a target eigenvalue, and identifying an eigenvalue range in which the target eigenvalue is located;

[0139] Step 505: Determine the output emotion type corresponding to the third voice segment according to the feature value range of the target feature value.

[0140] By performing feature fusion on the audio information and text information of the third voice segment, and subsequently performing eigenvalue calculation on the final fused feature vector, the target eigenvalue is obtained, and the eigenvalue range of the target eigenvalue is identified. According to the eigenvalue range of the target eigenvalue, the output emotion type corresponding to the third voice segment is determined, so that the intelligent voice customer service introduces emotional information when finally outputting the third voice segment.

[0141] Step 204: Control the current pronunciation object to output the third voice segment with the output emotion type, wherein the current pronunciation object includes intelligent voice customer service.

[0142] In this embodiment, the step of controlling the current pronunciation object to output the third voice segment with the output emotion type specifically includes: filtering out a target voice output node according to the output emotion type corresponding to the third voice segment; controlling the current pronunciation object to output the third voice segment through the target voice output node, wherein corresponding speaking tone, speaking pitch, sound intensity and sound speed are set for different voice output nodes in advance according to different emotion types.

[0143] The voice output method described in the present application is applied to the intelligent voice customer service consultation and answering scenario, which can enable the intelligent voice customer service's response sentence to fully consider the context information, and can not only obtain the sentence emotion information from the audio data, but also obtain the sentence emotion information from the text data, so that the output third voice segment is more emotionally rich and more in line with the actual voice consultation question and answer scenario. At the same time, in the intelligent voice customer service dialogue, the emotion type is introduced to avoid the output intelligent voice being too mechanical, thereby improving the customer's dialogue experience.

[0144] This application obtains a first voice segment, a second voice segment, and a third voice segment, wherein the first voice segment, the second voice segment, and the third voice segment are voice segments in the form of a question-and-answer dialogue; performs preprocessing to obtain the preprocessed first voice segment, the second voice segment, and the third voice segment; inputs the preprocessed first voice segment, the second voice segment, and the third voice segment into a multimodal information fusion model to predict the output emotion type of the third voice segment; and controls the current pronunciation object to output the third voice segment with the output emotion type. Applying the voice output method of this application to the intelligent voice customer service consultation and response scenario can make the response sentence of the intelligent voice customer service fully consider the context information, obtain the sentence emotion information from the audio data and the text data, and make the voice segment output by the intelligent voice customer service more emotional and more in line with the actual voice consultation and question-and-answer scenario. At the same time, in the intelligent voice customer service dialogue, the emotion type is introduced to avoid the output intelligent voice being too mechanical, thereby improving the customer's dialogue experience.

[0145] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0146] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0147] In an embodiment of the present application, by obtaining a first voice segment, a second voice segment and a third voice segment, the first voice segment, the second voice segment and the third voice segment are voice segments in the form of a question-and-answer dialogue; preprocessing is performed to obtain the preprocessed first voice segment, the second voice segment and the third voice segment; the preprocessed first voice segment, the second voice segment and the third voice segment are input into a multimodal information fusion model to predict the output emotion type of the third voice segment; and the current pronunciation object is controlled to output the third voice segment with the output emotion type. Applying the voice output method of the present application to the intelligent voice customer service consultation and response scenario can make the response statement of the intelligent voice customer service fully consider the context information, obtain the sentence emotion information from the audio data and the text data, so that the voice segment output by the intelligent voice customer service is more emotionally rich and more in line with the actual voice consultation question and answer scenario. At the same time, in the intelligent voice customer service dialogue, the emotion type is introduced to avoid the output intelligent voice being too mechanical, thereby improving the customer's dialogue experience.

[0148] Further references Figure 7 , as a response to the above Figure 2 The present application provides an embodiment of a speech output device, and the embodiment of the device is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0149] like Figure 7 As shown, the speech output device 700 described in this embodiment includes: a speech segment acquisition module 701, a speech segment preprocessing module 702, an output emotion prediction module 703 and a speech segment output module 704. Among them:

[0150] The voice segment acquisition module 701 is used to acquire a first voice segment, a second voice segment and a third voice segment, wherein the first voice segment is at least one sentence previously output by the current pronunciation object, the second voice segment is at least one sentence of the dialogue pronunciation object, and the third voice segment is a voice segment to be output by the current pronunciation object, and the first voice segment, the second voice segment and the third voice segment are continuous in time relationship;

[0151] A speech segment preprocessing module 702, configured to preprocess the first speech segment, the second speech segment and the third speech segment respectively to obtain the preprocessed first speech segment, the second speech segment and the third speech segment;

[0152] An output emotion prediction module 703 is used to input the preprocessed first voice segment, the second voice segment and the third voice segment into a preset multimodal information fusion model, and predict the output emotion type of the third voice segment according to the multimodal information fusion model;

[0153] The voice segment output module 704 is used to control the current pronunciation object to output the third voice segment with the output emotion type, wherein the current pronunciation object includes intelligent voice customer service.

[0154] This application obtains a first voice segment, a second voice segment, and a third voice segment, wherein the first voice segment, the second voice segment, and the third voice segment are voice segments in the form of a question-and-answer dialogue; performs preprocessing to obtain the preprocessed first voice segment, the second voice segment, and the third voice segment; inputs the preprocessed first voice segment, the second voice segment, and the third voice segment into a multimodal information fusion model to predict the output emotion type of the third voice segment; and controls the current pronunciation object to output the third voice segment with the output emotion type. Applying the voice output method of this application to the intelligent voice customer service consultation and response scenario can make the response sentence of the intelligent voice customer service fully consider the context information, obtain the sentence emotion information from the audio data and the text data, and make the voice segment output by the intelligent voice customer service more emotional and more in line with the actual voice consultation and question-and-answer scenario. At the same time, in the intelligent voice customer service dialogue, the emotion type is introduced to avoid the output intelligent voice being too mechanical, thereby improving the customer's dialogue experience.

[0155] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0156] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0157] To solve the above technical problems, the present application also provides a computer device. Figure 8 , Figure 8 This is a basic structural block diagram of the computer device in this embodiment.

[0158] The computer device 8 includes a memory 8a, a processor 8b, and a network interface 8c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 8 having components 8a-8c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0159] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0160] The memory 8a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 8a can be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 8a can also be an external storage device of the computer device 8, such as a plug-in hard disk equipped on the computer device 8, a smart memory card (Smart Med i aCard, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 8a can also include both the internal storage unit of the computer device 8 and its external storage device. In this embodiment, the memory 8a is generally used to store the operating system and various application software installed on the computer device 8, such as a computer-readable instruction of a voice output method, etc. In addition, the memory 8a can also be used to temporarily store various types of data that have been output or are to be output.

[0161] The processor 8b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chips in some embodiments. The processor 8b is generally used to control the overall operation of the computer device 8. In the present embodiment, the processor 8b is used to run the computer-readable instructions or process data stored in the memory 8a, such as the computer-readable instructions for running the speech output method.

[0162] The network interface 8c may include a wireless network interface or a wired network interface. The network interface 8c is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0163] The computer device proposed in this embodiment belongs to the field of speech processing technology and is applied to intelligent customer service speech output scenarios. This application obtains a first speech segment, a second speech segment and a third speech segment, and the first speech segment, the second speech segment and the third speech segment are speech segments in the form of a question-and-answer dialogue; performs preprocessing to obtain the preprocessed first speech segment, the second speech segment and the third speech segment; inputs the preprocessed first speech segment, the second speech segment and the third speech segment into a multimodal information fusion model to predict the output emotion type of the third speech segment; controls the current pronunciation object to output the third speech segment with the output emotion type. Applying the speech output method of this application to the intelligent voice customer service consultation and response scenario can make the response statement of the intelligent voice customer service fully consider the context information, obtain the sentence emotion information from the audio data and the text data, so that the speech segment output by the intelligent voice customer service is more emotional and more in line with the actual voice consultation question and answer scenario. At the same time, in the intelligent voice customer service dialogue, the emotion type is introduced to avoid the output intelligent voice being too mechanical, and improve the customer's dialogue experience.

[0164] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the voice output method as described above.

[0165] The computer-readable storage medium proposed in this embodiment belongs to the field of speech processing technology and is applied to the intelligent customer service speech output scenario. This application obtains the first voice segment, the second voice segment and the third voice segment, and the first voice segment, the second voice segment and the third voice segment are voice segments in the form of a question-and-answer dialogue; performs preprocessing to obtain the preprocessed first voice segment, the second voice segment and the third voice segment; inputs the preprocessed first voice segment, the second voice segment and the third voice segment into the multimodal information fusion model to predict the output emotion type of the third voice segment; controls the current pronunciation object to output the third voice segment with the output emotion type. Applying the speech output method of this application to the intelligent voice customer service consultation and response scenario can make the response statement of the intelligent voice customer service fully consider the context information, obtain the sentence emotion information from the audio data and the text data, so that the voice segment output by the intelligent voice customer service is more emotional and more in line with the actual voice consultation question and answer scenario. At the same time, in the intelligent voice customer service dialogue, the emotional type is introduced to avoid the output of the intelligent voice being too mechanical, and improve the customer's dialogue experience.

[0166] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0167] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A speech output method, characterized in that: The steps include: Acquire a first voice segment, a second voice segment, and a third voice segment, wherein the first voice segment is at least one sentence previously output by the current pronunciation object, the second voice segment is at least one sentence of the dialogue pronunciation object, and the third voice segment is a voice segment to be output by the current pronunciation object, and the first voice segment, the second voice segment, and the third voice segment are continuous in time relationship; Preprocessing the first speech segment, the second speech segment, and the third speech segment respectively to obtain the preprocessed first speech segment, the second speech segment, and the third speech segment; Inputting the preprocessed first speech segment, the second speech segment and the third speech segment into a preset multimodal information fusion model, and predicting the output emotion type of the third speech segment according to the multimodal information fusion model; The current pronunciation object is controlled to output the third voice segment with the output emotion type, wherein the current pronunciation object includes an intelligent voice customer service.

2. The speech output method according to claim 1, characterized in that: The step of preprocessing the first voice segment, the second voice segment and the third voice segment respectively to obtain the preprocessed first voice segment, the second voice segment and the third voice segment specifically includes: The first speech segment, the second speech segment and the third speech segment before preprocessing are used as the current speech segments to be processed in sequence; Performing audio and text separation processing on the current voice segment to be processed to obtain separated pairs of audio data and text data; Determine whether the audio data of the current voice segment to be processed exceeds a preset time length; If the audio data of the current voice segment to be processed does not exceed the preset time length, a silent segment is introduced to fill the audio data to obtain target audio data consistent with the preset time length; If the audio data of the current voice segment to be processed exceeds the preset time length, the audio data is intercepted and processed to obtain target audio data consistent with the preset time length; The target audio data and the text data are used as the preprocessed speech segment of the current speech segment to be processed.

3. The speech output method according to claim 2, characterized in that: The step of performing audio and text separation processing on the current voice segment to be processed to obtain separated paired audio data and text data specifically includes: Inputting the current speech segment to be processed into a preset ASR speech recognition model to obtain audio data and text data output by the ASR speech recognition model; Marking the audio data and text data with the distinguishing identification information of the current voice segment to be processed, and obtaining the marked paired audio data and text data; The step of introducing a silent segment to fill the audio data to obtain target audio data consistent with the preset time length specifically includes: The silent segment is spliced ​​with the endpoints of the audio data to obtain new audio data in the format of "silent segment + audio data" or "audio data + silent segment" as the target audio data, wherein the time length of the new audio data is consistent with the preset time length, and the time length of the silent segment is the difference between the time length of the audio data and the preset time length.

4. The speech output method according to claim 1, characterized in that: The multimodal information fusion model includes an audio feature encoding component, a text feature encoding component, a feature vector splicing and fusion component, and a speech output component. Before executing the step of inputting the preprocessed first speech segment, the second speech segment, and the third speech segment into the preset multimodal information fusion model, and predicting the output emotion type of the third speech segment according to the multimodal information fusion model, the method further includes: Obtain speech datasets of different emotion types; Preprocessing each speech segment in the speech data set to obtain a speech data set after preprocessing; Inputting the preprocessed speech data set into the multimodal information fusion model; By means of the audio feature encoding component, the audio feature of each speech segment in the speech data set is encoded to obtain audio feature encoding information corresponding to each speech segment; By means of the text feature encoding component, the text feature of each speech segment in the speech data set is encoded to obtain text feature encoding information corresponding to each speech segment; The feature vector splicing and fusion component is used to splice and fuse the audio feature coding information and the text feature coding information of the same speech segment to obtain a fused final feature vector; The final feature vectors corresponding to different speech segments of the same emotion type are clustered and sorted to obtain the final feature vector sets corresponding to different emotion types; According to the final feature vector sets corresponding to different emotion types, the feature value ranges corresponding to different emotion types are determined; According to the feature value ranges corresponding to different emotion types, an emotion type classifier is set in the fully connected layer corresponding to the speech output component, wherein setting the emotion type classifier means setting different speech output nodes for speech segments of different emotion types according to the feature value ranges in the fully connected layer corresponding to the speech output component.

5. The speech output method according to claim 4, characterized in that: The step of inputting the preprocessed first speech segment, the second speech segment and the third speech segment into a preset multimodal information fusion model, and predicting the output emotion type of the third speech segment according to the multimodal information fusion model specifically includes: By means of the audio feature encoding component, the audio features of the preprocessed first voice segment, the second voice segment and the third voice segment are respectively encoded to obtain audio feature encoding information corresponding to each voice segment; By means of the text feature encoding component, the text features of the preprocessed first voice segment, the second voice segment and the third voice segment are respectively encoded to obtain text feature encoding information corresponding to each voice segment; Using the feature vector splicing and fusion component, splicing and fusing the audio feature coding information and the text feature coding information of the third voice segment to obtain a fused final feature vector; Performing eigenvalue calculation on the final eigenvector after fusion to obtain a target eigenvalue, and identifying an eigenvalue range in which the target eigenvalue is located; The output emotion type corresponding to the third speech segment is determined according to the feature value range of the target feature value.

6. The speech output method according to claim 5, characterized in that: Before executing the step of using the feature vector splicing and fusion component to splice and fuse the audio feature coding information and the text feature coding information of the third voice segment to obtain a fused final feature vector, the method further includes: Introducing a contextual attention mechanism into the audio feature encoding component and the text feature encoding component; By using the context attention mechanism, the context dependency relationship between the audio feature encoding information of the third voice segment and the audio feature encoding information of the first voice segment and the second voice segment is calculated to obtain the audio context dependency relationship; and By using the context attention mechanism, the context dependency relationship between the text feature encoding information of the third voice segment and the text feature encoding information of the first voice segment and the second voice segment is calculated to obtain the text context dependency relationship; adding the audio context dependency as audio implicit information to the audio feature coding information of the third voice segment to update the audio feature coding information of the third voice segment; adding the text context dependency as text implicit information to the text feature coding information of the third voice segment to update the text feature coding information of the third voice segment; The feature vector splicing and fusion component includes a feature vector splicing and fusion component based on a cross-modal attention mechanism, and the step of using the feature vector splicing and fusion component to splice and fuse the audio feature encoding information and the text feature encoding information of the third voice segment to obtain the fused final feature vector specifically includes: The audio feature encoding information of the third voice segment is used as a query vector, and the text feature encoding information of the third voice segment is used as a key vector and a value vector, and the correlation between the audio feature encoding information and the text feature encoding information of the third voice segment is calculated according to the attention mechanism, and feature weights are assigned to the text feature encoding information of the third voice segment according to the correlation to obtain a first feature weight assignment matrix; Taking the text feature encoding information of the third voice segment as the Query vector, taking the audio feature encoding information of the third voice segment as the Key vector and the Value vector, and calculating the correlation between the text feature encoding information and the audio feature encoding information of the third voice segment according to the attention mechanism, and performing feature weight allocation on the audio feature encoding information of the third voice segment according to the correlation to obtain a second feature weight allocation matrix; The first feature weight allocation matrix and the second feature weight allocation matrix are concatenated and fused using a matrix fusion method to obtain the final feature vector.

7. The speech output method according to claim 4, characterized in that: The step of controlling the current pronunciation object to output the third voice segment with the output emotion type specifically includes: Filtering a target speech output node according to the output emotion type corresponding to the third speech segment; The current pronunciation object is controlled to output the third voice segment through the target voice output node, wherein corresponding speaking tone, speaking pitch, sound intensity and sound speed are respectively set for different voice output nodes according to different emotion types.

8. A speech output device, characterized in that: include: A voice segment acquisition module, used to acquire a first voice segment, a second voice segment and a third voice segment, wherein the first voice segment is at least one sentence previously output by the current pronunciation object, the second voice segment is at least one sentence of the dialogue pronunciation object, and the third voice segment is a voice segment to be output by the current pronunciation object, and the first voice segment, the second voice segment and the third voice segment are continuous in time relationship; A speech segment preprocessing module, used to preprocess the first speech segment, the second speech segment and the third speech segment respectively to obtain the preprocessed first speech segment, the second speech segment and the third speech segment; An output emotion prediction module is used to input the preprocessed first voice segment, the second voice segment and the third voice segment into a preset multimodal information fusion model, and predict the output emotion type of the third voice segment according to the multimodal information fusion model; A voice segment output module is used to control the current pronunciation object to output the third voice segment with the output emotion type, wherein the current pronunciation object includes intelligent voice customer service.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech output method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech output method according to any one of claims 1 to 7 are implemented.

Images