Voice response generation method and system

Abstract

The invention provides a voice response generation method and system. Relates to the technical field of language processing. The method comprises the steps that text features and acoustic features corresponding to voice signals are acquired, the text features are vector sets obtained by conducting character transcription processing on the voice signals, and the acoustic features are vector sets of acoustic attributes extracted from the voice signals; the confidence degree of a transcription result of the voice signal is obtained, the transcription result is a text character corresponding to the voice signal, and the confidence degree is used for representing the reliability degree of the transcription result; according to a learnable scaling factor and a confidence coefficient, the association weight of the text feature and the acoustic feature is dynamically adjusted, a target fusion feature is obtained, the learnable scaling factor is in positive correlation with the adjusted association weight, and the confidence coefficient is in positive correlation with the adjusted association weight; and obtaining a response result corresponding to the voice signal according to the target fusion feature. The accuracy of the response result can be improved.

Description

Technical Field

[0001] This application relates to the field of language processing technology, and in particular to a method and system for generating voice responses. Background Technology

[0002] In telemarketing scenarios, customers express their needs, emotions, or objections via voice. The system needs to quickly understand the customer's intentions (such as inquiries, complaints, etc.) and emotional state (such as anger, hesitation, etc.) and generate response scripts that are consistent with business objectives and emotions.

[0003] In related technologies, automatic speech recognition transcription results based on a single text modality are used for intent and emotion recognition. However, in telephone scenarios, there are complex factors such as noise interference in the voice signal, changes in speech rate, frequent pauses, and transcription errors, which lead to a high misjudgment rate in intent recognition, distortion in emotion judgment, and difficulty in generating responses that match the customer's actual emotions and needs. Summary of the Invention

[0004] This application provides a voice response generation method and system to improve the accuracy of voice responses.

[0005] Firstly, this application provides a method for generating a voice response, including:

[0006] The text features and acoustic features corresponding to the speech signal are obtained. The text features are a set of vectors obtained by transcribing the speech signal into text, and the acoustic features are a set of vectors of acoustic attributes extracted from the speech signal.

[0007] The confidence level of the transcription result of the speech signal is obtained. The transcription result is the text character corresponding to the speech signal. The confidence level is used to characterize the reliability of the transcription result.

[0008] The association weights of text features and acoustic features are dynamically adjusted based on the learnable scaling factor and confidence level to obtain the target fusion features. The learnable scaling factor is positively correlated with the adjusted association weights, and the confidence level is positively correlated with the adjusted association weights.

[0009] Based on the target fusion features, the response result corresponding to the speech signal is obtained.

[0010] In one possible implementation, the association weights of text features and acoustic features are dynamically adjusted based on a learnable scaling factor and confidence level to obtain the target fusion features, including:

[0011] Initial association weights are determined based on textual and acoustic features;

[0012] The initial association weights are adjusted using a learnable scaling factor to obtain the adjusted association weights;

[0013] Construct a confidence diagonal matrix based on the confidence scores, and perform a logarithmic transformation on the confidence diagonal matrix to obtain the confidence adjustment factor. The diagonal elements of the confidence diagonal matrix represent the confidence scores.

[0014] After adding the adjusted association weights and the confidence adjustment factor, normalization is performed to obtain the attention weight matrix.

[0015] The attention weight matrix and acoustic features are fused to obtain the initial fused features;

[0016] The initial fusion features and text features are fused together to obtain the target fusion features.

[0017] In one possible implementation, the text features and acoustic features corresponding to the speech signal are obtained, including:

[0018] The speech signal is transcribed into text to obtain the transcription result;

[0019] The transcription results are then processed into text features through text encoding.

[0020] The speech signal is processed by feature extraction to obtain feature extraction results, which include at least one of the following: Mel frequency cepstral coefficients, pitch curve, energy, speech rate, and pause duration;

[0021] The feature extraction results are aggregated based on the transcription results to obtain the acoustic representation;

[0022] The acoustic representation is processed by acoustic coding to obtain acoustic features.

[0023] In one possible implementation, the response result corresponding to the speech signal is obtained based on the target fusion features, including:

[0024] Multi-task recognition processing is performed on the target fusion features to obtain emotion recognition results and intent recognition results;

[0025] The text features, acoustic features, emotion recognition results, and intent recognition results are processed to obtain structured data;

[0026] The large language model is used to process the structured data to obtain candidate responses;

[0027] The response result is determined based on the candidate responses.

[0028] In one possible implementation, multi-task recognition processing is performed on the target fusion features to obtain emotion recognition results and intent recognition results, including:

[0029] The target fused features are pooled based on the confidence level to obtain the context vector;

[0030] The intent score vector is obtained by performing a fully connected process on the context vector through the intent prediction branch.

[0031] The sentiment score vector is obtained by performing a fully connected process on the context vector through the sentiment prediction branch.

[0032] The intent score vector is normalized to obtain the confidence level of the intent score vector. The intent corresponding to the vector with the highest confidence level in the intent score vector is determined as the intent recognition result.

[0033] The sentiment score vector is normalized to obtain the confidence level of the sentiment score vector. The sentiment corresponding to the vector with the highest confidence level is determined as the sentiment recognition result.

[0034] In one possible implementation, determining the response result based on candidate responses includes:

[0035] Determine the log probability of the candidate response; the log probability is used to characterize the confidence level of the candidate response.

[0036] The context vector and candidate response are projected into the same normalized embedding space to determine the similarity between the context vector and the candidate response;

[0037] A consistency score is determined based on similarity.

[0038] Candidate responses are ranked according to their consistency scores, and the response result is determined based on the ranking.

[0039] In one possible implementation, before calling a large language model to process the structured data and obtain candidate responses, the method also includes:

[0040] The initial model is used to process the training samples to obtain the predicted context vector. The predicted context vector represents the fusion result of the text features and acoustic features of the training samples. The training samples include annotation information, which is used to indicate the true sentiment, true intention and expected response.

[0041] The initial model is used to process the prediction context vector to obtain the prediction results, which are used to indicate the predicted sentiment and predicted intent.

[0042] The prediction results and prediction context vectors are processed using a large language model to obtain the predicted candidate responses;

[0043] Determine the target response based on the predicted candidate responses;

[0044] The target loss function is determined based on the prediction context vector, the prediction result, the target response, and the training samples.

[0045] The initial model is trained under supervision using the target loss function to obtain the target model.

[0046] In one possible implementation, the target loss function is determined based on the prediction result, the target response, and the training samples, including:

[0047] Determine the first loss function based on the predicted intent and the actual intent;

[0048] The second loss function is determined based on the predicted sentiment and the actual sentiment.

[0049] The third loss function is determined based on the target response and the expected response;

[0050] The fourth loss function is determined based on the similarity between the target response and the predicted context vector;

[0051] The sum of the first loss function, the second loss function, the third loss function, and the fourth loss function is determined as the target loss function.

[0052] Secondly, this application provides a voice response generation system, comprising:

[0053] The first acquisition module is used to acquire the text features and acoustic features corresponding to the speech signal. The text features are a set of vectors obtained by transcribing the speech signal into text, and the acoustic features are a set of vectors of acoustic attributes extracted from the speech signal.

[0054] The second acquisition module is used to acquire the confidence level of the transcription result of the speech signal. The transcription result is the text characters corresponding to the speech signal, and the confidence level is used to characterize the reliability of the transcription result.

[0055] The processing module is used to dynamically adjust the association weights of text features and acoustic features based on a learnable scaling factor and a confidence level to obtain the target fusion features. The learnable scaling factor is positively correlated with the adjusted association weights, and the confidence level is positively correlated with the adjusted association weights.

[0056] The determination module is used to obtain the response result corresponding to the speech signal based on the target fusion features.

[0057] In one possible implementation, the processing module is specifically used for:

[0058] Initial association weights are determined based on textual and acoustic features;

[0059] The initial association weights are adjusted using a learnable scaling factor to obtain the adjusted association weights;

[0060] Construct a confidence diagonal matrix based on the confidence scores, and perform a logarithmic transformation on the confidence diagonal matrix to obtain the confidence adjustment factor. The diagonal elements of the confidence diagonal matrix represent the confidence scores.

[0061] After adding the adjusted association weights and the confidence adjustment factor, normalization is performed to obtain the attention weight matrix.

[0062] The attention weight matrix and acoustic features are fused to obtain the initial fused features;

[0063] The initial fusion features and text features are fused together to obtain the target fusion features.

[0064] In one possible implementation, the first acquisition module is specifically used for:

[0065] The speech signal is transcribed into text to obtain the transcription result;

[0066] The transcription results are then processed into text features through text encoding.

[0067] The speech signal is processed by feature extraction to obtain feature extraction results, which include at least one of the following: Mel frequency cepstral coefficients, pitch curve, energy, speech rate, and pause duration;

[0068] The feature extraction results are aggregated based on the transcription results to obtain the acoustic representation;

[0069] The acoustic representation is processed by acoustic coding to obtain acoustic features.

[0070] In one possible implementation, the determination module is specifically used for:

[0071] Multi-task recognition processing is performed on the target fusion features to obtain emotion recognition results and intent recognition results;

[0072] The text features, acoustic features, emotion recognition results, and intent recognition results are processed to obtain structured data;

[0073] The large language model is used to process the structured data to obtain candidate responses;

[0074] The response result is determined based on the candidate responses.

[0075] In one possible implementation, the determining module is also used for:

[0076] The target fused features are pooled based on the confidence level to obtain the context vector;

[0077] The intent score vector is obtained by performing a fully connected process on the context vector through the intent prediction branch.

[0078] The sentiment score vector is obtained by performing a fully connected process on the context vector through the sentiment prediction branch.

[0079] The intent score vector is normalized to obtain the confidence level of the intent score vector. The intent corresponding to the vector with the highest confidence level in the intent score vector is determined as the intent recognition result.

[0080] The sentiment score vector is normalized to obtain the confidence level of the sentiment score vector. The sentiment corresponding to the vector with the highest confidence level is determined as the sentiment recognition result.

[0081] In one possible implementation, the determining module is also used for:

[0082] Determine the log probability of the candidate response; the log probability is used to characterize the confidence level of the candidate response.

[0083] The context vector and candidate response are projected into the same normalized embedding space to determine the similarity between the context vector and the candidate response;

[0084] A consistency score is determined based on similarity.

[0085] Candidate responses are ranked according to their consistency scores, and the response result is determined based on the ranking.

[0086] In one possible implementation, the voice response generation system is also used for:

[0087] The initial model is used to process the training samples to obtain the predicted context vector. The predicted context vector represents the fusion result of the text features and acoustic features of the training samples. The training samples include annotation information, which is used to indicate the true sentiment, true intention and expected response.

[0088] The initial model is used to process the prediction context vector to obtain the prediction results, which are used to indicate the predicted sentiment and predicted intent.

[0089] The prediction results and prediction context vectors are processed using a large language model to obtain the predicted candidate responses;

[0090] Determine the target response based on the predicted candidate responses;

[0091] The target loss function is determined based on the prediction context vector, the prediction result, the target response, and the training samples.

[0092] The initial model is trained under supervision using the target loss function to obtain the target model.

[0093] In one possible implementation, the voice response generation system is also used for:

[0094] Determine the first loss function based on the predicted intent and the actual intent;

[0095] The second loss function is determined based on the predicted sentiment and the actual sentiment.

[0096] The third loss function is determined based on the target response and the expected response;

[0097] The fourth loss function is determined based on the similarity between the target response and the predicted context vector;

[0098] The sum of the first loss function, the second loss function, the third loss function, and the fourth loss function is determined as the target loss function.

[0099] Thirdly, this application provides an electronic device, including: a memory and a processor;

[0100] The memory stores the instructions that the computer executes;

[0101] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0102] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible embodiments of the first aspect.

[0103] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0104] The speech response generation method and system provided in this application extract textual and acoustic features of speech signals and obtain the confidence level of the transcription results of the speech signals. A learnable scaling factor and the confidence level are used to dynamically adjust the association weights of the textual and acoustic features to generate target fusion features. Response results are then generated based on the target fusion features. The learnable scaling factor is positively correlated with the adjusted association weights, allowing the scaling factor to adjust the association requirements of textual and acoustic features under different application scenarios. The confidence level is also positively correlated with the adjusted association weights, allowing the confidence level to suppress the interference of text transcription errors on the target fusion features and improve the accuracy of the response results. Attached Figure Description

[0105] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0106] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0107] Figure 2A flowchart illustrating the voice response generation method provided in this application embodiment;

[0108] Figure 3 for Figure 2 A flowchart illustrating the specific implementation of step S103 in the illustrated embodiment;

[0109] Figure 4 for Figure 2 A flowchart illustrating the specific implementation of step S104 in the illustrated embodiment;

[0110] Figure 5 A schematic flowchart illustrating the model training method provided in this application embodiment;

[0111] Figure 6 A schematic diagram of the structure of the voice response generation system provided in this application;

[0112] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0113] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0114] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0115] Figure 1 This is a schematic diagram of an application scenario provided in an embodiment of this application, such as... Figure 1 As shown, it includes a client 10 and a server 11, and a communication connection between the client 10 and the server 11.

[0116] Client 10 is a device used by users for voice interaction in outbound call scenarios, such as smartphones, landline phones, smart voice terminals, etc. Client 10 can collect the voice signals generated by users during the call, such as the customer's inquiry voice in a customer service call. The voice signal contains information such as the user's speech content, tone, and background noise. Client 10 transmits the collected voice signal to server 11.

[0117] Server 11 is a device with computing capabilities. Server 11 is used to receive voice signals, perform text recognition, intent recognition and emotion recognition on the voice signals, generate response text based on the recognition results, convert the response text into voice form to obtain a response signal, and feed the response signal back to client 10. Client 10 receives the response signal to realize automatic voice response.

[0118] Traditional response text generation schemes rely on Automatic Speech Recognition (ASR) transcription results for intent and emotion recognition. However, telephone scenarios are plagued by complex factors such as speech signal noise interference, speech rate variations, frequent pauses, and ASR transcription errors, leading to high false positive rates in intent recognition, distorted emotion judgment, and generated responses that fail to match the user's actual emotions and needs. Furthermore, users may convey implicit information (such as hesitation or anxiety) through acoustic features like intonation, pitch, and pauses during conversations, information that cannot be directly reflected in the text modality. Therefore, integrating acoustic features of speech with textual information to achieve finer-grained multimodal alignment and joint modeling is crucial for improving the accuracy of intent recognition, the robustness of emotion judgment, and the quality of generated responses.

[0119] In related technologies, speech information is converted into text through ASR and used as input for subsequent intent recognition and emotion recognition. Intent and emotion are identified through an independent text classification model. Responses are generated based on fixed templates or large language models (LLMs). Text and acoustic features are fused through simple concatenation or parallel input to generate response responses.

[0120] The aforementioned technologies rely excessively on ASR text and fail to effectively utilize emotional cues in acoustic features (such as pitch and speech rate); intent recognition and emotion recognition are independent modules, lacking joint training and alignment constraints, and thus cannot enhance each other; the prompts for input LLM are natural language descriptions, lacking structured design with word-level timestamps, confidence scores, and acoustic projections, resulting in unstable emotional consistency and intent matching of generated responses; and candidate responses are ranked based solely on language model probabilities without combining multimodal similarity (text and acoustics) for re-ranking, making it impossible to select the optimal response when ASR errors or emotional cues are important.

[0121] Based on this, embodiments of this application provide a voice response generation method. This method extracts textual and acoustic features from a speech signal and obtains the confidence level of the transcription result. It then dynamically adjusts the association weights of the textual and acoustic features using a learnable scaling factor and the confidence level to generate a target fusion feature. Finally, it generates a response result based on the target fusion feature. The learnable scaling factor is positively correlated with the adjusted association weights, allowing adjustment of the association requirements between textual and acoustic features in different application scenarios. The confidence level is also positively correlated with the adjusted association weights, enabling suppression of transcription errors from interfering with the target fusion feature and improving the accuracy of the response result.

[0122] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0123] Figure 2 This is a flowchart illustrating the voice response generation method provided in an embodiment of this application. The execution subject of this method can be a computing device or a processor located within the computing device. For example, the computing device can be a server. For example, the processor in the computing device can be the central processing unit (CPU) of a server. For ease of understanding, the following description will use a computing device as the execution subject. Figure 2 As shown, the method includes:

[0124] S101. Obtain the text features and acoustic features corresponding to the speech signal.

[0125] Speech signals are user voices captured by sound acquisition devices, such as user inquiries captured through microphones, telephone lines, or recording equipment. Speech signals include information such as speech content, tone of voice, and background noise. Text features are a set of vectors obtained by transcribing the speech signal into text, while acoustic features are a set of vectors extracting acoustic attributes from the speech signal, such as speech rate, average energy, and pitch range.

[0126] The speech response generation method provided in this application can be implemented using a target model. The target model is a model trained using training samples and a loss function. The target model may include multiple modules, each performing a different function, such as a text feature extraction module, an acoustic feature extraction module, a multimodal encoder (including a text encoder and an acoustic encoder), a cross-modal attention fusion module, a multi-task recognition module, a projection and alignment module, a structured data generation module, and a rearrangement module. The functions and specific implementation methods of each module are described in detail in the following steps.

[0127] For example, obtaining text features corresponding to a speech signal includes: performing text transcription processing on the speech signal to obtain a transcription result; and performing text encoding processing on the transcription result to obtain text features.

[0128] Specifically, the text feature extraction module of the target model can be used to transcribe the speech signal, generating a transcription result. Specifically, the text feature extraction module uses ASR technology to transcribe the speech signal word by word, that is, to identify individual words or consecutive word groups in the speech signal and generate the corresponding text, thus obtaining the transcription result. The transcription result can be represented as... Here, n represents the number of words. For example, the transcription result for the speech signal "I want to inquire about the price" could be {I want, inquire, a, price}.

[0129] Optionally, the transcription results may also include a timestamp and confidence score for each word group. The timestamp indicates the start and end times of pronunciation for that word group in the speech signal, and the confidence score indicates the reliability of the word recognition result. The transcription result for the i-th word can be represented as follows: Here, `word` represents a single word or a continuous phrase, `start` indicates the time point at which the word begins to be pronounced in the question, `end` represents the end timestamp, and `conf` represents the confidence level of the word, with a value ranging from [0,1], where 0 indicates completely unreliable and 1 indicates completely reliable. For example, for the phrase "I think", the transcription result is: .

[0130] Furthermore, discrete text words can be transformed into continuous, semantically information-rich low-dimensional vectors using a text encoder within the multimodal coding module, yielding text features. The text encoder is a pre-trained language model or deep learning network, such as the Transformer model or Bidirectional Encoder Representations from Transformers (BERT). Text features can be represented as... The dimension is n×d, where n is the number of word groups and d is the output dimension.

[0131] For example, obtaining acoustic features corresponding to a speech signal includes: performing feature extraction processing on the speech signal to obtain feature extraction results, wherein the feature extraction results include at least one of the following: Mel frequency cepstral coefficients, pitch curve, energy, speech rate, and pause duration; performing aggregation processing on the feature extraction results based on the transcription results to obtain an acoustic representation; and performing acoustic encoding processing on the acoustic representation to obtain acoustic features.

[0132] The acoustic feature extraction module of the target model can be used to extract features from the speech signal to obtain the feature extraction results. Then, the acoustic encoder in the multimodal encoder can be used to aggregate and encode the feature extraction results to obtain the acoustic features.

[0133] Specifically, the acoustic feature extraction module splits the speech signal into frames (e.g., 20ms / frame, with a 10ms overlap), extracts the acoustic features of each frame, obtains the feature extraction results, and concatenates the feature extraction results into a frame-level vector. The feature extraction results can include Mel-Frequency Cepstral Coefficients (MFCC), pitch curve, energy, speech rate, and pause duration. MFCC is the core acoustic feature extracted by simulating the characteristics of human hearing (sensitive to low frequencies and insensitive to high frequencies); the pitch curve is the fundamental frequency of the vocal cord vibration, reflecting the speaker's tone (e.g., high or low) and emotion (e.g., a higher pitch when angry); energy is the signal strength (sum of squares of amplitude) of the speech frame, reflecting the volume (e.g., high energy when speaking loudly, low energy when speaking softly); speech rate is the speed of speech, i.e., the number of words (or syllables) pronounced per unit time; and pause duration is the length of the silent period between adjacent words in the speech (excluding normal speech intervals, referring only to obvious pauses).

[0134] The aggregation process aligns frame-level vectors by word. Based on the timestamps of each word group in the transcription results, it aggregates the frame-level vectors within the time range of that word, assigning one vector to each word group to obtain the acoustic representation of that word group. The acoustic representation can be expressed as... .

[0135] Furthermore, deep feature extraction and dimensionality transformation can be performed on the acoustic representation using an acoustic encoder to generate acoustic features with the same dimensionality as the text features. The acoustic encoder is a pre-trained language model or deep learning network, such as a Transformer model, a hybrid model of CNN and Transformer, etc. The acoustic encoder can extract local features and temporal correlations from the acoustic representation, mapping them to d dimensions to obtain acoustic features. These acoustic features can be represented as... The dimensions are the same as those of the text features.

[0136] In the above method, by performing text transcription and text encoding on the speech signal, the semantic information in the speech can be accurately converted into text features that the model can process, providing a reliable text information foundation for subsequent multimodal fusion. By extracting multi-dimensional acoustic attributes, including Mel frequency cepstral coefficients and pitch curves, and combining the extracted acoustic features with the transcription results for aggregation processing, the acoustic information in the speech signal can be fully captured and the positional alignment of acoustic features with text information can be achieved. By performing acoustic encoding processing on the aggregated acoustic representation, the representational ability of acoustic features can be improved, adapting to the needs of subsequent correlation weight adjustment and feature fusion.

[0137] S102. Obtain the confidence level of the transcription results of the speech signal.

[0138] The transcription result is the text characters corresponding to the speech signal, and the confidence level is used to characterize the reliability of the transcription result.

[0139] For example, the confidence level can be obtained directly by calling the confidence output interface of the existing ASR system through the text feature extraction module. The confidence level ranges from [0,1], where 0 represents completely unreliable and 1 represents completely reliable. For example, the confidence level is low when there is strong background noise or the user's pronunciation is not standard, and high when there is no background noise and the user's pronunciation is standard and clear.

[0140] S103. Dynamically adjust the association weights of text features and acoustic features based on the learnable scaling factor and confidence level to obtain the target fusion features.

[0141] Among them, the learnable scaling factor is positively correlated with the adjusted association weights, and the confidence level is also positively correlated with the adjusted association weights. The learnable scaling factor adjusts the association weights to adapt to the association requirements of text features and acoustic features in different scenarios, and the confidence level can be used to adjust the association weights to suppress the interference of text transcription errors on the target fusion features.

[0142] The target fusion features can be determined using a cross-modal attention fusion module. This module fuses textual and acoustic features, adjusting the association weights based on a learnable scaling factor and confidence level during the fusion process. The importance of textual and acoustic features varies across different interaction scenarios, and the learnable scaling factor automatically adapts to these needs. For example, in customer service scenarios, emotional information is more important, so the weight of acoustic features needs to be increased; in technical query scenarios, semantic information is more important, so the weight of textual features needs to be increased.

[0143] The association weights are a weight matrix used to measure the interaction strength between textual and acoustic features, determining the contribution ratio of the two types of features during fusion. The learnable scaling factor is a parameter obtained during training the target model, used to adjust the relative importance of textual and acoustic features. The target fusion feature is the sum vector feature obtained after weighted fusion of textual and acoustic features, combining the accuracy of textual semantics with the integrity of acoustic sentiment.

[0144] For example, the initial association weights can be multiplied element-wise using a learnable scaling factor to adjust the overall association strength between the two types of features. Then, the adjusted association weights can be fused using confidence to obtain the final association weights. Finally, the target fused feature can be obtained based on the final association weights.

[0145] The Transformer attention mechanism can be used to determine target fusion features, for example, such as Figure 3 As shown, step S103 can be achieved by the following steps:

[0146] S1031. Determine the initial association weights based on text features and acoustic features.

[0147] The initial association weight S can be determined by the following formula:

[0148]

[0149] in, T represents text features. Acoustic characteristics, and Here, the weight matrix is ​​learnable and is determined during the training of the target model. This is used to linearly transform text features into the query vector Q required for attention calculation. This is used to linearly transform acoustic features into the key vector K required for attention computation. This is a scaling factor used for scaling. Scaling is performed to prevent Too large.

[0150] S1032. Adjust the initial association weights using a learnable scaling factor to obtain the adjusted association weights.

[0151] For example, the range of the learnable scaling factor can be adjusted to [0,1] using the sigmoid activation function, and then the adjusted learnable scaling factor is multiplied by the initial association weights to obtain the adjusted association weights. This process can be represented as follows: ,in For learnable parameters, It is the sigmoid activation function. This indicates element-wise multiplication.

[0152] S1033. Construct a confidence diagonal matrix based on the confidence level, and perform a logarithmic transformation on the confidence diagonal matrix to obtain the confidence adjustment factor, where the diagonal elements of the confidence diagonal matrix are the confidence levels.

[0153] Each word group corresponds to a confidence level. For multiple confidence levels of a speech signal, they are converted into a diagonal matrix, with the diagonal elements representing the confidence levels and the rest being 0, resulting in a confidence diagonal matrix. Then, take the logarithm of each element in the confidence diagonal matrix, i.e., log(C), to obtain the confidence adjustment factor.

[0154] S1034. After adding the adjusted association weights and the confidence adjustment factor, normalize the results to obtain the attention weight matrix.

[0155] For example, the process can be represented as Here, softmax is a normalization function used to transform the value range of each element in the attention weight matrix to [0,1]. For the i-th word group, .

[0156] The confidence adjustment factor is used to prevent low-confidence phrases from actively associating with unreliable acoustic features, thus avoiding false propagation.

[0157] S1035. The attention weight matrix and acoustic features are fused to obtain the initial fused features.

[0158] The initial fusion feature Attn can be determined by the following formula:

[0159]

[0160] in, , Acoustic characteristics, Here, the weight matrix is ​​learnable and is determined during the training of the target model. Used to linearly convert acoustic features into a value vector V.

[0161] S1036. Perform fusion processing on the initial fusion features and text features to obtain the target fusion features.

[0162] Target fusion features It can be determined by the following formula:

[0163]

[0164] Among them, FFN is a feedforward fully connected network, which is used to fuse the initial fusion features and text features into a deep semantic representation, namely the target fusion feature, through two layers of linear mapping and nonlinear activation. The target fusion feature contains context-dependent information.

[0165] In the above method, the initial association weights are adjusted by a learnable scaling factor, which enables the association weights to dynamically adapt to speech and text features and acoustic features globally, thus meeting the fusion requirements of multimodal information in different scenarios. By constructing a confidence diagonal matrix and performing logarithmic transformation, as well as normalizing the weights after addition, the reliance on unreliable words is reduced when the confidence is low or the speech contains noise, avoiding the amplification of erroneous information. This significantly improves the stability and accuracy of intent recognition and emotion recognition in actual telephone call environments.

[0166] S104. Based on the target fusion features, obtain the response result corresponding to the speech signal.

[0167] The response result is a reply statement generated based on the content of the speech signal. It can be obtained by a multi-task recognition module that identifies user intent and emotion based on the target fusion result. The structured data generation module integrates the data output from the previous modules to generate structured data, which is then input into a large language model. The large language model generates multiple reply statements that conform to user intent and emotion. These multiple reply statements are returned to the reordering module of the target model through an interface. The reordering module determines the response result from the multiple reply statements.

[0168] For example, such as Figure 4 As shown, step S104 can be achieved by the following steps:

[0169] S1041. Perform multi-task recognition processing on the target fusion features to obtain emotion recognition results and intent recognition results.

[0170] A multi-task recognition module can be used to process the fused features of the target to obtain sentiment recognition and intent recognition results. This module can include an intent prediction branch and a sentiment prediction branch. The intent prediction branch is used to obtain the intent recognition result, and the sentiment prediction branch is used to obtain the sentiment recognition result. The intent recognition result indicates what the user wants to do, such as querying or complaining; the sentiment recognition result characterizes the user's emotions, such as dissatisfaction, happiness, or neutrality.

[0171] For example, multi-task recognition processing is performed on the target fusion features to obtain emotion recognition results and intent recognition results, including: pooling the target fusion features according to the confidence level to obtain a context vector; performing fully connected processing on the context vector through the intent prediction branch to obtain an intent score vector; performing fully connected processing on the context vector through the sentiment prediction branch to obtain a sentiment score vector; normalizing the intent score vector to obtain the confidence level of the intent score vector, and determining the intent corresponding to the vector with the highest confidence level in the intent score vector as the intent recognition result; normalizing the sentiment score vector to obtain the confidence level of the sentiment score vector, and determining the sentiment corresponding to the vector with the highest confidence level in the sentiment score vector as the emotion recognition result.

[0172] The context vector is a global feature vector obtained by weighted pooling of the target fused features. This vector contains global information about the semantics, acoustics, and cross-modal correlations of the speech signal. The weights used in the weighted pooling are the confidence scores of the transcription results. The context vector u can be determined by the following formula:

[0173]

[0174] Where c represents the confidence level of the transcription result, and h represents the target fusion feature.

[0175] The intent score vector, output by the intent prediction branch, indicates the user's intent and reflects the prediction strength of each category; a higher value indicates a more likely category. The sentiment score vector, output by the sentiment prediction branch, indicates the user's sentiment and reflects the prediction strength of each sentiment; a higher value indicates a more likely sentiment. For example, when the intent categories include query, feedback, complaint, consultation, and other (5 types), the intent score vector is [3.2, 0.8, 0.5, 1.1, 0.3], indicating that the user's inquiry is most likely a query. When the sentiment categories include neutral, dissatisfied, and happy (3 types), the sentiment score vector is [2.8, 0.4, 0.6], indicating that the user's sentiment is most likely neutral.

[0176] The intent prediction branch obtains the intent score vector by performing fully connected processing on the context vector. Specifically, the input context vector first passes through a fully connected layer to double its dimensionality, then through a ReLU activation function, and finally through another fully connected layer to reduce its dimensionality to the target dimension, resulting in the final intent score vector. The fully connected processing process in the sentiment prediction branch is similar to that in the intent prediction branch, and will not be described in detail here.

[0177] Furthermore, the intent score vector is normalized using the softmax function to obtain its confidence level, and the intent category corresponding to the highest confidence level is determined as the intent recognition result. For example, after normalizing the intent score vector [3.2, 0.8, 0.5, 1.1, 0.3], the obtained confidence level is [0.92, 0.03, 0.02, 0.02, 0.01], and the intent recognition result is query.

[0178] Furthermore, the softmax function is used to normalize the sentiment score vector to obtain the confidence level of the sentiment score vector, and the sentiment category corresponding to the highest confidence level is determined as the sentiment recognition result. For example, after normalizing the sentiment score vector [2.8, 0.4, 0.6], the obtained confidence level is [0.88, 0.07, 0.05], and the sentiment recognition result is neutral.

[0179] In the above method, pooling the target fusion features based on confidence level yields a context vector. This pooling process enhances the contribution of high-confidence information and suppresses interference from low-confidence information, improving the accuracy of the context vector in representing the core speech information. The intent prediction branch performs fully connected processing on the context vector to obtain an intent score vector, accurately extracting intent-related features and achieving quantitative representation of intent information. The sentiment prediction branch performs fully connected processing on the context vector to obtain a sentiment score vector, effectively capturing the sentiment semantic information within the context vector and completing the quantitative conversion of sentiment information. By normalizing the score vector and selecting the result with the highest confidence level as the recognition result, the most reliable intent and sentiment conclusions can be selected, ensuring the accuracy and stability of the multi-task recognition results.

[0180] S1042. Process the text features, acoustic features, emotion recognition results, and intent recognition results to obtain structured data.

[0181] The structured data generation module can process text features, acoustic features, sentiment recognition results, and intent recognition results to obtain structured data. This structured data, in standardized JSON format, is integrated from these data and used as input to a large language model, enabling the model to generate response statements based on this structured data.

[0182] Optionally, the structured data also includes historical dialogue data from the most recent K rounds, which helps the large language model generate responses that are closer to the expected responses in multi-round dialogue scenarios.

[0183] Furthermore, the structured data retains timestamps and confidence levels, allowing large language models to use these timestamps and confidence levels to convert structured data into natural language instruction fragments, thus obtaining prompt words.

[0184] S1043. Call the large language model to process the structured data and obtain candidate responses.

[0185] The large language model is a trained model used to generate candidate responses to speech signals. These candidate responses are natural language text that aligns with the user's intent and emotion; multiple candidate responses are allowed. After generating candidate responses, the large language model returns them to the target model via an interface.

[0186] S1044. Determine the response result based on the candidate responses.

[0187] A projection and alignment module can project candidate responses and context vectors into the same normalized embedding space to calculate the similarity between the context vectors and candidate responses. Then, a rearrangement module determines the candidate response that best matches the speech signal from among multiple candidate responses based on the similarity, thus obtaining the response result. Specifically, the cosine similarity between the candidate response and the context vector can be calculated, and the candidate response with the highest cosine similarity is determined as the response result.

[0188] For example, determining the response result based on candidate responses includes: determining the log probability of the candidate responses, whereby the log probability is used to characterize the confidence level of the candidate responses; projecting the context vector and candidate responses into the same normalized embedding space to determine the similarity between the context vector and the candidate responses; determining a consistency score based on the similarity; ranking the candidate responses based on the consistency score; and determining the response result based on the ranking result.

[0189] Specifically, for each candidate response, its log probability is calculated. The log probability of a candidate response can be expressed as... This indicates that, given the context vector u, the large language model generates candidate responses. The probability of a candidate response. The log probability of the candidate response is directly returned by the interface of the large language model.

[0190] To calculate the similarity between the context vector and the candidate response, both need to be projected into the same normalized embedding space. This projection can be achieved using a Multilayer Perceptron (MLP). First, a high-dimensional linear transformation is performed in the first layer of the MLP to uncover deep semantic relationships among features. Then, a ReLU activation function is used to introduce non-linearity. Next, a second layer of the MLP compresses the complex high-dimensional features to the target dimension, balancing feature representation and computational cost. Finally, L2 normalization ensures that all vectors have a magnitude of 1. For example, for the input context vector and candidate response, the first layer of the MLP increases the dimension to 1024, followed by an activation function, and then the second layer of the MLP reduces the dimension to 256. The output 256-dimensional features are then L2 normalized to obtain normalized embedding vectors. These normalized embedding vectors include the context embedding vector corresponding to the context vector and the candidate response embedding vector corresponding to the candidate response. All vectors are in the same normalized embedding space, allowing for direct similarity calculation. The similarity between the context vector and the candidate response can also be determined by calculating cosine similarity.

[0191] The consistency score is used to quantify the degree of matching between the candidate response embedding vector and the context embedding vector. The consistency score can be determined by the following formula:

[0192]

[0193] Where r represents the candidate response. For candidate response embedding vectors, Embedded vectors for context, This indicates that the logarithmic probabilities of candidate responses have been normalized. Normalization methods include min-max normalization, etc. These are learnable parameters that are automatically determined during the training of the target model. Their values ​​range from [0,1] and are used to adjust the contribution weight of similarity in the consistency score. A higher value indicates that similarity is given more weight in the consistency score calculation. The smaller the value, the smaller the impact of similarity.

[0194] Furthermore, a rearrangement formula can be determined based on the consistency score. A rearrangement score can then be calculated using this formula. Candidate responses are then sorted in descending order of their rearrangement scores, and the candidate response with the highest rearrangement score (ranked first) is selected as the response result. The rearrangement formula can be expressed as:

[0195]

[0196] Where S(r) is the rearrangement score and C(u,r) is the consistency score. These are learnable parameters that are automatically determined during the training of the target model. Their values ​​range from [0,1] and are used to adjust the priority of the log probability and consistency score of candidate responses during rearrangement. A larger value indicates that the rearrangement prioritizes logarithmic probability, giving preference to candidate responses that are fluent and semantically coherent. A lower score indicates a greater emphasis on consistency, prioritizing candidate responses that closely match the customer's emotions and intentions.

[0197] In the above method, by determining the log probability of candidate responses, the confidence level of candidate responses can be quantitatively represented, providing a basic and reliable basis for subsequent screening of high-quality responses; by projecting the context vector and candidate responses into the same normalized embedding space and determining their similarity, the degree of matching between candidate responses and core speech information (including semantics and emotion) can be accurately measured; by fusing similarity and the log probability of candidate responses to determine the consistency score, and by considering both log probability and consistency score in the rearrangement formula, the final response result maintains language fluency and better reflects the customer's emotions and intentions, reducing responses with mismatched tone or deviating from intent, thereby improving user experience and business conversion rate.

[0198] The speech response generation method provided in this application extracts textual and acoustic features from a speech signal and obtains the confidence level of the transcription result. It then dynamically adjusts the association weights of the textual and acoustic features using a learnable scaling factor and the confidence level to generate a target fusion feature. Finally, it generates a response result based on the target fusion feature. The learnable scaling factor is positively correlated with the adjusted association weights, allowing adjustment of the association requirements between textual and acoustic features in different application scenarios. The confidence level is also positively correlated with the adjusted association weights, enabling suppression of transcription errors from interfering with the target fusion feature and improving the accuracy of the response result.

[0199] Optionally, embodiments of this application also provide a model training method for training an initial model using a loss function to obtain a target model, enabling the target model to perform tasks such as... Figure 2 The method for generating voice responses is shown.

[0200] Figure 5 This is a flowchart illustrating the model training method provided in the embodiments of this application, as shown below. Figure 5 As shown, the method includes:

[0201] S201. Use the initial model to process the training samples to obtain the predicted context vector.

[0202] The initial model has the same structure as the target model, including a text feature extraction module, an acoustic feature extraction module, a multimodal encoder (including a text encoder and an acoustic encoder), a cross-modal attention fusion module, a multi-task recognition module, a projection and alignment module, a structured data generation module, and a rearrangement module. The specific implementation of each module is similar to that of the target model; for the same implementation, please refer to [link to relevant documentation]. Figure 2 The embodiments shown are not described in detail here.

[0203] The training samples are a set of samples including multiple speech signals, used to train the initial model. The training samples also include annotation information, which is used to indicate the true emotion, true intention and expected response.

[0204] The predicted context vector represents the fusion result of textual and acoustic features of the training samples, corresponding to the context vector u in the target model. Similarly, the process of determining the predicted context vector includes: transcribing the training samples using a text feature extraction module to generate transcription results; extracting features from the training samples using an acoustic feature extraction module to obtain feature extraction results; encoding the transcription results and feature extraction results separately using a multimodal encoder to obtain textual and acoustic features; determining the confidence level of the transcription results using the text feature extraction module; adjusting the association weights of the textual and acoustic features using a cross-modal attention fusion module and performing fusion processing to obtain the target fused features; and processing the target fused features using a multi-task recognition module to obtain the predicted context vector. For the specific implementation details of each step, please refer to [link to details]. Figure 2 The illustrated embodiment.

[0205] S202. Use the initial model to process the prediction context vector to obtain the prediction result.

[0206] The prediction results are used to indicate predicted sentiment and predicted intent. The prediction context vector is processed using the initial model to obtain the prediction results, which include: obtaining the predicted intent through the intent prediction branch of the multi-task recognition module, and obtaining the predicted sentiment through the sentiment prediction branch of the multi-task recognition module. The predicted sentiment includes a sentiment score vector and its corresponding confidence level, and the predicted intent includes an intent score vector and its corresponding confidence level.

[0207] S203. Use a large language model to process the prediction results and prediction context vectors to obtain the predicted candidate responses.

[0208] The structured data generation module of the initial model integrates the prediction results and prediction context vectors to generate structured data. This structured data is then input into the large language model, which generates prompt words based on the structured data and then generates predicted candidate responses based on the prompt words. The predicted candidate responses correspond to... Figure 2In the embodiments, candidate responses are predicted to be natural language response text that conforms to the user intent and sentiment indicated by the training samples.

[0209] S204. Determine the target response based on the predicted candidate responses.

[0210] The target response is the predicted candidate response that best matches the training samples. Determining the target response based on the predicted candidate responses involves: projecting the predicted candidate response and the context vector into the same normalized embedding space using a projection and alignment module to calculate the similarity between the predicted context vector and the predicted candidate response; then, using a re-ranking module, determining a consistency score based on the similarity, determining a re-ranking formula based on the consistency score, obtaining a re-ranking score, and selecting the predicted candidate response with the highest re-ranking score as the target response.

[0211] S205. Determine the target loss function based on the prediction results, target response, and training samples.

[0212] The objective loss function is used to optimize the model parameters of the initial model to improve the accuracy of intent recognition and sentiment recognition, thereby improving the quality of the predicted response text. The model parameters of the initial model include a learnable scaling factor. Weight matrix in cross-modal attention fusion module Learnable parameters in the rearrangement module wait.

[0213] For example, determining the target loss function based on the predicted context vector, the predicted result, the target response, and the training samples includes: determining a first loss function based on the predicted intent and the true intent; determining a second loss function based on the predicted sentiment and the true sentiment; determining a third loss function based on the target response and the expected response; determining a fourth loss function based on the similarity between the target response and the predicted context vector; and determining the sum of the first loss function, the second loss function, the third loss function, and the fourth loss function as the target loss function.

[0214] The target loss function L can be determined by the following formula:

[0215]

[0216] in, The first loss function is used to measure the gap between the predicted intent and the true intent. Minimizing the first loss function improves the accuracy of intent recognition. The second loss function is used to measure the gap between the predicted sentiment and the real sentiment. Minimizing the second loss function improves the accuracy of sentiment recognition. The third loss function is used to measure the semantic gap between the target response and the expected response. Minimizing the third loss function improves the relevance, fluency, and accuracy of the target response. The fourth loss function measures the spatial consistency between the predicted context vector and the target response. By minimizing the fourth loss function, we ensure that the target response not only matches semantically but also aligns with the timestamp, thereby improving the relevance of the target response under short sentences and hesitant tones. To adjust the loss weights, they can be adaptively adjusted during training or preset according to the business scenario. The sum of is 1.

[0217] The fourth loss function can be determined by the following formula:

[0218]

[0219] Where sim represents cosine similarity. The value representing temperature is typically between 0.07 and 0.1, used to control the smoothness of the similarity distribution. K is the number of predicted candidate responses. (In the molecule) The cosine similarity between the target response and the predicted context vector is given by the denominator. To predict the cosine similarity between the predicted candidate responses (excluding the target response) and the predicted context vector.

[0220] S206. Use the target loss function to perform supervised training on the initial model to obtain the target model.

[0221] By minimizing the objective loss function, the parameters of the initial model are continuously optimized until a preset training threshold is reached, thus obtaining the target model. The training threshold is a pre-set, dynamically adjustable condition used to indicate the termination of the model training process, such as a similarity between the target response and the expected response exceeding 0.98.

[0222] The model training method provided in this application optimizes the initial model by jointly using the first to fourth loss functions. This makes the predicted context vector not only easier to classify but also closer to the text representation of the correct response in the vector space, thereby improving the ranking probability of the correct (semantic / sentimental consistent) response among the predicted candidate responses. Especially in telephone outbound calling scenarios, the fourth loss function can significantly improve recall and consistency under short sentences and hesitant tones, thus distinguishing it from the conventional multi-task approach of only training in parallel.

[0223] Figure 6 A schematic diagram of the structure of the voice response generation system provided in this application is shown below. Figure 6 As shown, the voice response generation system 20 provided in this embodiment includes:

[0224] The first acquisition module 21 is used to acquire the text features and acoustic features corresponding to the speech signal. The text features are a set of vectors obtained by transcribing the speech signal into text, and the acoustic features are a set of vectors of acoustic attributes extracted from the speech signal.

[0225] The second acquisition module 22 is used to acquire the confidence level of the transcription result of the speech signal. The transcription result is the text characters corresponding to the speech signal, and the confidence level is used to characterize the reliability of the transcription result.

[0226] Processing module 23 is used to dynamically adjust the association weights of text features and acoustic features according to the learnable scaling factor and confidence level to obtain target fusion features, wherein the learnable scaling factor is positively correlated with the adjusted association weights, and the confidence level is positively correlated with the adjusted association weights.

[0227] The determination module 24 is used to obtain the response result corresponding to the speech signal based on the target fusion features.

[0228] In one possible implementation, processing module 23 is specifically used for:

[0229] Initial association weights are determined based on textual and acoustic features;

[0230] The initial association weights are adjusted using a learnable scaling factor to obtain the adjusted association weights;

[0231] Construct a confidence diagonal matrix based on the confidence scores, and perform a logarithmic transformation on the confidence diagonal matrix to obtain the confidence adjustment factor. The diagonal elements of the confidence diagonal matrix represent the confidence scores.

[0232] After adding the adjusted association weights and the confidence adjustment factor, normalization is performed to obtain the attention weight matrix.

[0233] The attention weight matrix and acoustic features are fused to obtain the initial fused features;

[0234] The initial fusion features and text features are fused together to obtain the target fusion features.

[0235] In one possible implementation, the first acquisition module 21 is specifically used for:

[0236] The speech signal is transcribed into text to obtain the transcription result;

[0237] The transcription results are then processed into text features through text encoding.

[0238] The speech signal is processed by feature extraction to obtain feature extraction results, which include at least one of the following: Mel frequency cepstral coefficients, pitch curve, energy, speech rate, and pause duration;

[0239] The feature extraction results are aggregated based on the transcription results to obtain the acoustic representation;

[0240] The acoustic representation is processed by acoustic coding to obtain acoustic features.

[0241] In one possible implementation, the determining module 24 is specifically used for:

[0242] Multi-task recognition processing is performed on the target fusion features to obtain emotion recognition results and intent recognition results;

[0243] The text features, acoustic features, emotion recognition results, and intent recognition results are processed to obtain structured data;

[0244] The large language model is used to process the structured data to obtain candidate responses;

[0245] The response result is determined based on the candidate responses.

[0246] In one possible implementation, the determining module 24 is also used for:

[0247] The target fused features are pooled based on the confidence level to obtain the context vector;

[0248] The intent score vector is obtained by performing a fully connected process on the context vector through the intent prediction branch.

[0249] The sentiment score vector is obtained by performing a fully connected process on the context vector through the sentiment prediction branch.

[0250] The intent score vector is normalized to obtain the confidence level of the intent score vector. The intent corresponding to the vector with the highest confidence level in the intent score vector is determined as the intent recognition result.

[0251] The sentiment score vector is normalized to obtain the confidence level of the sentiment score vector. The sentiment corresponding to the vector with the highest confidence level is determined as the sentiment recognition result.

[0252] In one possible implementation, the determining module 24 is also used for:

[0253] Determine the log probability of the candidate response; the log probability is used to characterize the confidence level of the candidate response.

[0254] The context vector and candidate response are projected into the same normalized embedding space to determine the similarity between the context vector and the candidate response;

[0255] A consistency score is determined based on similarity.

[0256] Candidate responses are ranked according to their consistency scores, and the response result is determined based on the ranking.

[0257] In one possible implementation, the voice response generation system 20 is also used for:

[0258] The initial model is used to process the training samples to obtain the predicted context vector. The predicted context vector represents the fusion result of the text features and acoustic features of the training samples. The training samples include annotation information, which is used to indicate the true sentiment, true intention and expected response.

[0259] The initial model is used to process the prediction context vector to obtain the prediction results, which are used to indicate the predicted sentiment and predicted intent.

[0260] The prediction results and prediction context vectors are processed using a large language model to obtain the predicted candidate responses;

[0261] Determine the target response based on the predicted candidate responses;

[0262] The target loss function is determined based on the prediction context vector, the prediction result, the target response, and the training samples.

[0263] The initial model is trained under supervision using the target loss function to obtain the target model.

[0264] In one possible implementation, the voice response generation system 20 is also used for:

[0265] Determine the first loss function based on the predicted intent and the actual intent;

[0266] The second loss function is determined based on the predicted sentiment and the actual sentiment.

[0267] The third loss function is determined based on the target response and the expected response;

[0268] The fourth loss function is determined based on the similarity between the target response and the predicted context vector;

[0269] The sum of the first loss function, the second loss function, the third loss function, and the fourth loss function is determined as the target loss function.

[0270] The voice response generation system provided in this embodiment can execute the voice response generation method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0271] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device 30 provided in this embodiment includes at least one processor 301 and a memory 302. Optionally, the electronic device 30 further includes a communication component 303. The processor 301, memory 302, and communication component 303 are connected via a bus.

[0272] In a specific implementation, at least one processor 301 executes computer execution instructions stored in memory 302, causing at least one processor 301 to execute the above-described voice response generation method embodiment.

[0273] The specific implementation process of processor 301 can be found in the above method embodiments, and its implementation principle and technical effect are similar, so it will not be repeated here. In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0274] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0275] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0276] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described voice response generation method.

[0277] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described voice response generation method.

[0278] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0279] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0280] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0281] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0282] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0283] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory, magnetic disks, or optical disks.

[0284] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0285] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method of generating a voice response, characterized by, The method comprises: obtaining text features and acoustic features corresponding to a voice signal, the text features being a vector set obtained by performing text transcription processing on the voice signal, and the acoustic features being a vector set of acoustic attributes extracted from the voice signal; obtaining a confidence degree of a transcription result of the voice signal, the transcription result being text characters corresponding to the voice signal, and the confidence degree being used to represent a reliability degree of the transcription result; dynamically adjusting associated weights of the text features and the acoustic features according to a learnable scaling factor and the confidence degree, to obtain target fusion features, wherein the learnable scaling factor is positively correlated with the adjusted associated weights, and the confidence degree is positively correlated with the adjusted associated weights; obtaining a response result corresponding to the voice signal according to the target fusion features.

2. The voice response generation method according to claim 1, characterized by, The method comprises: determining initial associated weights according to the text features and the acoustic features; adjusting the initial associated weights using the learnable scaling factor to obtain adjusted associated weights; constructing a confidence degree diagonal matrix according to the confidence degree, and performing logarithmic conversion on the confidence degree diagonal matrix to obtain a confidence degree adjustment factor, wherein diagonal elements of the confidence degree diagonal matrix are the confidence degree; adding the adjusted associated weights and the confidence degree adjustment factor, and performing normalization processing to obtain an attention weight matrix; performing fusion processing on the attention weight matrix and the acoustic features to obtain initial fusion features; performing fusion processing on the initial fusion features and the text features to obtain the target fusion features.

3. The voice response generation method according to claim 1, characterized by, The method comprises: performing text transcription processing on the voice signal to obtain a transcription result; performing text encoding processing on the transcription result to obtain the text features; performing feature extraction processing on the voice signal to obtain a feature extraction result, the feature extraction result comprising at least one of the following: mel frequency cepstral coefficient, pitch contour, energy, speech rate, and pause duration; performing aggregation processing on the feature extraction result according to the transcription result to obtain an acoustic representation; performing acoustic encoding processing on the acoustic representation to obtain the acoustic features.

4. The voice response generation method of claim 1, wherein, The method comprises: performing multi-task recognition processing on the target fusion features to obtain emotion recognition results and intent recognition results; processing the text features, the acoustic features, the emotion recognition results, and the intent recognition results to obtain structured data; calling a large language model to process the structured data to obtain a candidate reply; determining the response result according to the candidate reply.

5. The voice response generation method according to claim 4, characterized by, The method comprises: performing pooling processing on the target fusion features according to the confidence degree to obtain a context vector; The context vector is fully connected by an intention prediction branch to obtain an intention score vector; The context vector is fully connected by an emotion prediction branch to obtain an emotion score vector; The intention score vector is normalized to obtain the confidence of the intention score vector, and the intention corresponding to the vector with the highest confidence in the intention score vector is determined as the intention recognition result; The emotion score vector is normalized to obtain the confidence of the emotion score vector, and the emotion corresponding to the vector with the highest confidence in the emotion score vector is determined as the emotion recognition result.

6. The voice response generation method according to claim 5, characterized by, The determination of the response result according to the candidate reply comprises: determining the logarithmic probability of the candidate reply, which represents the confidence of the candidate reply; projecting the context vector and the candidate reply into the same normalized embedding space to determine the similarity between the context vector and the candidate reply; determining a consistency score according to the similarity; sorting the candidate reply according to the consistency score, and determining the response result according to the sorting result.

7. The voice response generation method according to claim 4, characterized by, Before the candidate reply is obtained by processing the structured data by calling the large language model, the method further comprises: processing the training sample using an initial model to obtain a predicted context vector, wherein the predicted context vector represents the fusion result of the text features and acoustic features of the training sample, the training sample includes annotation information, and the annotation information is used to indicate the real emotion, the real intention and the expected reply; processing the predicted context vector using the initial model to obtain a prediction result, wherein the prediction result is used to indicate the predicted emotion and the predicted intention; processing the prediction result and the predicted context vector using a large language model to obtain a predicted candidate reply; determining a target reply according to the predicted candidate reply; determining a target loss function according to the predicted context vector, the prediction result, the target reply and the training sample; supervising and training the initial model using the target loss function to obtain a target model.

8. The voice response generation method according to claim 7, characterized by, The determination of the target loss function according to the prediction result, the target reply and the training sample comprises: determining a first loss function according to the predicted intention and the real intention; determining a second loss function according to the predicted emotion and the real emotion; determining a third loss function according to the target reply and the expected reply; determining a fourth loss function according to the similarity between the target reply and the predicted context vector; determining the sum of the first loss function, the second loss function, the third loss function and the fourth loss function as the target loss function.

9. A voice response generation system, characterized by comprises: a first acquisition module, configured to acquire text features and acoustic features corresponding to a voice signal, wherein the text features are a vector set obtained by performing text transcription processing on the voice signal, and the acoustic features are a vector set of acoustic properties extracted from the voice signal; A second obtaining module is configured to obtain a confidence degree of a transcription result of the speech signal, the transcription result being text characters corresponding to the speech signal, and the confidence degree being used to represent a reliable degree of the transcription result. A processing module is configured to dynamically adjust an associated weight of the text feature and the acoustic feature according to a learnable scaling factor and the confidence degree, to obtain a target fusion feature, wherein the learnable scaling factor is positively correlated with the adjusted associated weight, and the confidence degree is positively correlated with the adjusted associated weight. A determining module is configured to obtain a response result corresponding to the speech signal according to the target fusion feature.

10. An electronic device, comprising: comprising: a memory, a processor; the memory stores computer-executed instructions; the processor executes the computer-executed instructions stored in the memory, so that the processor executes the method in any one of claims 1-8.

Images