An interactive response method and system based on large language model

Through an interactive response method based on a large language model, combined with speech processing and sentiment analysis, intelligently generate digital human interaction text, solving the problem of the inability to understand complex queries and lack of emotional understanding in the existing technology, and achieving more natural and efficient user interaction Experience.

CN118467703BActive Publication Date: 2024-11-29SUYUAN CULTURE TECH CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202410686193.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-30
Publication Date
2024-11-29
Estimated Expiration
2044-05-30

AI Technical Summary

Technical Problem

Existing interactive response methods cannot fully understand the user's complex queries or context, and lack the ability to understand the user's emotions and emotions, resulting in poor user experience.

Method used

Using an interactive response method based on a large language model, by detecting the interaction between the user and the digital human object, generating an interaction window, receiving the user's interactive voice signal, and performing speech processing and text semantic coding, integrating text semantics and emotional waveform semantic features , intelligently generate digital human interactive text.

Benefits of technology

Deeply understand the user's query intention and context information, identify the user's emotional state, and adjust the emotional color of the reply, making the interaction process more natural and improving the overall user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118467703B_ABST
    Figure CN118467703B_ABST
Patent Text Reader

Abstract

The present application discloses an interactive response method and system based on a large language model, which generates an interactive window in response to detecting a user's action of determining to interact with a first digital human object, and receives an interactive voice signal input by the user in the interactive window, and uses artificial intelligence-based voice signal processing and recognition technology to perform voice processing and text semantic encoding on the interactive voice signal, so as to intelligently generate digital human interactive text based on the fusion features of the text semantics and emotional waveform semantics of the interactive voice signal and play it in the interactive window. In this way, the user's query intention and context information can be deeply understood, and the user's emotional state can be identified, and the emotional color of the reply can be adjusted accordingly, making the interaction process more natural, thereby improving the overall user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of intelligent interaction, and more specifically, to an interactive response method and system based on a large language model. Background Art

[0002] Interactive response is a communication method in which users can exchange information and feedback in real time. They can have a two-way dialogue with human users, understand users' questions or instructions, and provide corresponding answers or perform corresponding tasks.

[0003] However, current interactive response methods cannot fully understand users' complex queries or contexts, resulting in unsatisfactory user experience. In addition, some existing interactive responses usually respond through a single user voice, but lack the ability to understand user emotions and feelings, which may lead to poor performance in processing user emotions.

[0004] Therefore, an interactive response method based on a large language model is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides an interactive response method and system based on a large language model, which generates an interactive window in response to detecting the action of the user determining to interact with the first digital human object, and receives the interactive voice signal input by the user in the interactive window, and uses artificial intelligence-based voice signal processing and recognition technology to perform voice processing and text semantic encoding on the interactive voice signal, so as to intelligently generate digital human interactive text based on the fusion characteristics of the text semantics and emotional waveform semantics of the interactive voice signal and play it in the interactive window. In this way, the user's query intention and context information can be deeply understood, and the user's emotional state can be identified, and the emotional color of the reply can be adjusted accordingly, making the interaction process more natural, thereby improving the overall user experience.

[0006] According to one aspect of the present application, there is provided an interactive response method based on a large language model, comprising:

[0007] In response to detecting an action of the user determining to interact with the first digital human object, generating an interaction window;

[0008] In the interactive window, receiving an interactive voice signal input by the user;

[0009] Performing speech recognition on the interactive speech signal to obtain an interactive speech text recognition result;

[0010] Performing semantic coding on the interactive speech text recognition result to obtain an interactive speech text semantic coding feature vector;

[0011] Extracting emotion pattern features from the interactive speech signal to obtain an interactive speech emotion pattern waveform semantic representation vector;

[0012] Input the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector into a feature value-by-feature value granularity interactive fusion module to obtain an interactive text semantic-interactive emotion fusion representation vector as an interactive text semantic-interactive emotion fusion representation feature;

[0013] Based on the interactive text semantics-interactive emotion fusion representation feature, the digital human interactive text is obtained and played.

[0014] In the above-mentioned interactive response method based on a large language model, the interactive speech text recognition result is semantically encoded to obtain an interactive speech text semantic encoding feature vector, including: performing word segmentation processing on the interactive speech text recognition result and then passing it through an interactive speech text semantic encoder based on a BERT model to obtain the interactive speech text semantic encoding feature vector.

[0015] In the above-mentioned interactive response method based on a large language model, the emotional pattern features of the interactive speech signal are extracted to obtain an interactive speech emotional pattern waveform semantic representation vector, including: inputting the interactive speech signal into an emotional pattern feature extractor based on a convolutional neural network model to obtain the interactive speech emotional pattern waveform semantic representation vector.

[0016] In the above-mentioned interactive response method based on a large language model, the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector are input into a feature value-by-feature value granularity interactive fusion module to obtain an interactive text semantic-interactive emotion fusion representation vector as an interactive text semantic-interactive emotion fusion representation feature, including: calculating a threshold vector based on the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector; and, using the threshold values ​​of each position in the threshold vector, calculating the weighted sum of the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector to obtain the interactive text semantic-interactive emotion fusion representation vector.

[0017] In the above-mentioned interactive response method based on a large language model, a threshold vector is calculated based on the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector, including: cascading the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector to obtain an interactive speech cascade feature vector; and, first multiplying the interactive speech cascade feature vector with a transformation matrix, then adding the obtained feature vector and the bias vector by position, and then inputting the obtained feature vector into a si gmo id activation function to obtain the threshold vector.

[0018] In the above-mentioned interactive response method based on a large language model, the weighted sum of the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector is calculated with the threshold value of each position in the threshold vector to obtain the interactive text semantic-interactive emotion fusion representation vector, including: using the eigenvalue of each position in the interactive speech text semantic coding feature vector as the exponent of the natural constant to calculate the product between the exponential function value with the natural constant as the base and one minus the corresponding threshold value to obtain an interactive speech text semantic coding feature vector composed of multiple interactive speech text semantic feature values; using the eigenvalue of each position in the interactive speech emotion pattern waveform semantic representation vector as the exponent of the natural constant to calculate the product between the exponential function value with the natural constant as the base and the corresponding threshold value to obtain an interactive speech emotion pattern waveform semantic coding feature vector composed of multiple interactive speech emotion pattern waveform semantic feature values; and, adding the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic coding feature vector by position to obtain the interactive text semantic-interactive emotion fusion representation vector.

[0019] In the above-mentioned interactive response method based on a large language model, the range of each threshold value in the threshold vector is greater than or equal to zero and less than or equal to one.

[0020] In the above-mentioned interactive response method based on a large language model, based on the interactive text semantic-interactive emotion fusion representation feature, a digital human interactive text is obtained and played, including: inputting the interactive text semantic-interactive emotion fusion representation vector into an interactive content generator based on AI GC to obtain a digital human interactive text; and converting the digital human interactive text into a voice signal and playing it in the interactive window.

[0021] According to another aspect of the present application, there is provided an interactive response system based on a large language model, comprising:

[0022] An interaction window generating module, configured to generate an interaction window in response to detecting an action of a user determining to interact with the first digital human object;

[0023] An interactive voice signal receiving module, used for receiving an interactive voice signal input by the user in the interactive window;

[0024] An interactive voice signal recognition module, used to perform voice recognition on the interactive voice signal to obtain an interactive voice text recognition result;

[0025] An interactive speech text semantic coding module, used for semantically coding the interactive speech text recognition result to obtain an interactive speech text semantic coding feature vector;

[0026] An emotion pattern feature extraction module is used to extract emotion pattern features from the interactive speech signal to obtain an interactive speech emotion pattern waveform semantic representation vector;

[0027] A semantic emotion fusion module, used for inputting the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector into a feature value-by-feature value granularity interactive fusion module to obtain an interactive text semantic-interactive emotion fusion representation vector as an interactive text semantic-interactive emotion fusion representation feature;

[0028] The digital human interactive text generation module is used to obtain the digital human interactive text and play it based on the interactive text semantic-interactive emotion fusion representation feature.

[0029] In the above-mentioned interactive response system based on a large language model, the emotion pattern feature extraction module is used to: input the interactive speech signal into an emotion pattern feature extractor based on a convolutional neural network model to obtain a semantic representation vector of the interactive speech emotion pattern waveform.

[0030] Compared with the prior art, the interactive response method and system based on a large language model provided by the present application generates an interactive window in response to detecting the action of the user determining to interact with the first digital human object, and receives the interactive voice signal input by the user in the interactive window, and uses the voice signal processing and recognition technology based on artificial intelligence to perform voice processing and text semantic encoding on the interactive voice signal, so as to intelligently generate the digital human interactive text based on the fusion characteristics of the text semantics and emotional waveform semantics of the interactive voice signal and play it in the interactive window. In this way, the user's query intention and context information can be deeply understood, and the user's emotional state can be identified, and the emotional color of the reply can be adjusted accordingly, making the interaction process more natural, thereby improving the overall user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0032] Figure 1 It is a flowchart of an interactive response method based on a large language model according to an embodiment of the present application.

[0033] Figure 2 Schematic diagram of the architecture of an interactive response method based on a large language model according to an embodiment of the present application.

[0034] Figure 3 The present invention is a flowchart of inputting the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector into the feature value granularity interactive fusion module in the interactive response method based on the large language model according to the embodiment of the present application to obtain the interactive text semantic-interactive emotion fusion representation vector as the interactive text semantic-interactive emotion fusion representation feature.

[0035] Figure 4 The present invention is a flowchart for calculating a threshold vector based on the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector in an interactive response method based on a large language model according to an embodiment of the present application.

[0036] Figure 5 In the interactive response method based on a large language model according to an embodiment of the present application, the threshold values ​​of each position in the threshold vector are used to calculate the weighted sum of the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector to obtain the interactive text semantic-interactive emotion fusion representation vector.

[0037] Figure 6 It is a block diagram of an interactive response system based on a large language model according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, but rather these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0039] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0040] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0041] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0042] Interactive response is a communication method in which users can exchange information and feedback in real time. They can have a two-way dialogue with human users, understand users' questions or instructions, and provide corresponding answers or perform corresponding tasks.

[0043] However, current interactive response methods cannot fully understand users' complex queries or contexts, resulting in unsatisfactory user experience. In addition, some existing interactive responses usually respond through a single user voice, but lack the ability to understand user emotions and feelings, which may lead to poor performance in processing user emotions.

[0044] It should be understood that large language models (LLMs) refer to large machine learning models classified by size and complexity, which are specially designed to process and generate natural language text. With the continuous development and updating of large language models, the application scenarios of language models have been greatly enriched, enabling them to perform well in a variety of interactive response tasks.

[0045] Based on this, the present application proposes an interactive response method based on a large language model, which generates an interactive window in response to detecting the action of the user determining to interact with the first digital human object, and in the interactive window, receives the interactive voice signal input by the user, and uses artificial intelligence-based voice signal processing and recognition technology to perform voice processing and text semantic encoding on the interactive voice signal, so as to intelligently generate digital human interactive text based on the fusion features of the text semantics and emotional waveform semantics of the interactive voice signal and play it in the interactive window. In this way, the user's query intention and context information can be deeply understood, and the user's emotional state can be identified, and the emotional color of the reply can be adjusted accordingly, making the interaction process more natural, thereby improving the overall user experience.

[0046] Figure 1 It is a flowchart of an interactive response method based on a large language model according to an embodiment of the present application. Figure 2 FIG. 1 is a schematic diagram of the architecture of an interactive response method based on a large language model according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to the embodiment of the present application, the interactive response method based on the large language model includes: S110, in response to detecting the action of the user determining to interact with the first digital human object, generating an interactive window; S120, in the interactive window, receiving the interactive voice signal input by the user; S130, performing voice recognition on the interactive voice signal to obtain an interactive voice text recognition result; S140, performing semantic encoding on the interactive voice text recognition result to obtain an interactive voice text semantic encoding feature vector; S150, performing emotion pattern feature extraction on the interactive voice signal to obtain an interactive voice emotion pattern waveform semantic representation vector; S160, inputting the interactive voice text semantic encoding feature vector and the interactive voice emotion pattern waveform semantic representation vector into the feature value granularity interactive fusion module to obtain an interactive text semantic-interactive emotion fusion representation vector as an interactive text semantic-interactive emotion fusion representation feature; and, S170, based on the interactive text semantic-interactive emotion fusion representation feature, obtaining the digital human interactive text and playing it.

[0047] In step S110, in response to detecting the user's action of determining to interact with the first digital human object, an interaction window is generated. It should be understood that an interaction window can be created in this way to allow the user to interact with the digital human object naturally and intuitively.

[0048] In step S120, in the interaction window, the interactive voice signal input by the user is received. Accordingly, considering that the interactive voice signal input by the user refers to the sound emitted when the user communicates with the digital human object through voice, these sound signals contain the user's query, instruction or any other information of the interaction with the digital human. Based on this, in order to provide a more natural and intuitive interactive experience, in the technical solution requested by our company, in the interaction window, the interactive voice signal input by the user is received, that is, by analyzing the voice signal, the user's intention and needs can be identified to understand the user's emotional state, better understand and respond to the user's needs, and thus provide corresponding services or information.

[0049] In step S130, speech recognition is performed on the interactive voice signal to obtain an interactive voice text recognition result. It should be understood that, considering that the interactive voice signal reflects the user's questions, such as questions about the life, thoughts, achievements of historical figures or questions about a historical event, these questions contain rich semantic information, and these semantic information plays an important role and is helpful for the generation of digital human interactive text. Based on this, in order to better understand and process the user's intentions and needs expressed in the interactive voice signal in the future, in the technical solution of the present application, speech recognition is performed on the interactive voice signal to obtain an interactive voice text recognition result, which can convert the voice signal into text form and is conducive to the subsequent semantic understanding and analysis of the interactive text and the accuracy of the text generation of the digital human interactive dialogue.

[0050] In step S140, the interactive speech text recognition result is semantically encoded to obtain an interactive speech text semantic encoding feature vector. Specifically, in an embodiment of the present application, the interactive speech text recognition result is semantically encoded to obtain an interactive speech text semantic encoding feature vector, including: the interactive speech text recognition result is subjected to word segmentation processing and then passed through an interactive speech text semantic encoder based on a BERT model to obtain the interactive speech text semantic encoding feature vector. Accordingly, considering that the interactive speech text recognition result contains a large amount of user questions and intention semantic information, and there is a semantic correlation between the contexts between these user questions and intentions. Therefore, in order to more clearly understand the semantic information between the contexts contained in the interactive speech text recognition results, in the technical solution of the present application, the interactive speech text recognition result is semantically encoded to obtain an interactive speech text semantic encoding feature vector. In particular, in a specific embodiment of the present application, the interactive speech text recognition result is subjected to word segmentation processing and then passed through an interactive speech text semantic encoder based on a BERT model to obtain the interactive speech text semantic encoding feature vector. In detail, the interactive speech text recognition result is segmented to divide the entire interactive speech text recognition result into multiple words containing interactive speech local recognition results, and then the segmented interactive speech text recognition result is input into an interactive speech text semantic encoder based on the BERT model to capture the semantic information between each word, thereby improving the understanding of user intentions and needs.

[0051] In step S150, the emotion pattern feature extraction is performed on the interactive voice signal to obtain the interactive voice emotion pattern waveform semantic representation vector. Specifically, in an embodiment of the present application, the emotion pattern feature extraction is performed on the interactive voice signal to obtain the interactive voice emotion pattern waveform semantic representation vector, including: inputting the interactive voice signal into an emotion pattern feature extractor based on a convolutional neural network model to obtain the interactive voice emotion pattern waveform semantic representation vector. It should be understood that, considering that the interactive voice signal expresses the hidden emotional feature information of the user when asking questions, such as whether the user's speaking attitude is friendly or serious, the user's speaking emotions, etc., these user-user emotional feature information provides a huge basis and support for generating digital human interactive text that is more in line with the user's emotional color. And considering that the convolutional neural network model has good features and capture extraction capabilities in speech signal processing, implicit emotion-related features can be extracted from the speech signal. Therefore, in the technical solution of the present application, the interactive speech signal is input into an emotion pattern feature extractor based on a convolutional neural network model to capture and mine the emotional semantic feature information expressed by the user hidden in the interactive speech signal, thereby obtaining an interactive speech emotion pattern waveform semantic representation vector containing richer emotional feature information.

[0052] In step S160, the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector are input into the feature value granularity interactive fusion module to obtain the interactive text semantic-interactive emotion fusion representation vector as the interactive text semantic-interactive emotion fusion representation feature. Figure 3 The flow chart of the interactive response method based on the large language model according to the embodiment of the present application is to input the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector into the feature value granularity interactive fusion module to obtain the interactive text semantic-interactive emotion fusion representation vector as the interactive text semantic-interactive emotion fusion representation feature. Specifically, in the embodiment of the present application, Figure 3As shown, the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector are input into the feature value granularity interactive fusion module to obtain the interactive text semantic-interactive emotion fusion representation vector as the interactive text semantic-interactive emotion fusion representation feature, including: S210, calculating the threshold vector based on the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector; and, S220, using the threshold values ​​of each position in the threshold vector, calculating the weighted sum of the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector to obtain the interactive text semantic-interactive emotion fusion representation vector. Accordingly, considering that the interactive speech text semantic encoding feature vector captures the semantic content in the interactive speech text, including the meaning of the text, key information, etc., this helps to better understand the meaning and intention of the user input text, thereby improving the interactive effect and accuracy of the dialogue. The interactive speech emotion pattern waveform semantic representation vector reflects the emotional content in the interactive speech, including the speaker's emotions, emotional tendencies, etc., which can better identify and analyze the user's emotional state, thereby achieving more intelligent emotion recognition and feedback. Therefore, in order to comprehensively represent the fusion features of the interactive speech text semantic coding features and the interactive speech emotion pattern waveform semantic representation features to comprehensively improve the interactive effect, emotion recognition and user experience of the conversation, in the technical solution of the present application, the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector are input into the feature value granularity interactive fusion module to obtain the interactive text semantic-interactive emotion fusion representation vector. Specifically, the eigenvalue-by-eigenvalue granularity interactive fusion module obtains a threshold vector through the cascade vector of the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector, so as to perform feature selection when integrating semantic and emotional information, that is, to determine which features are the most important and effective for the final fusion representation, thereby screening out the most discriminative and influential features for the task, and then weightedly fuses the eigenvalues ​​of each position in the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector by the threshold value of each position in the threshold vector. In this way, the importance of the granularity of feature values ​​at different positions can be dynamically adjusted in a more fine-grained manner, so that the contribution of each feature can be more flexibly considered in the fusion process, so as to better capture the relationship between the interactive speech semantics and the interactive speech emotion information, thereby improving the expressive power of the interactive text semantics-interactive emotion fusion representation vector.

[0053] Figure 4The flowchart is a flow chart of calculating a threshold vector based on the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector in the interactive response method based on the large language model according to the embodiment of the present application. More specifically, in the embodiment of the present application, Figure 4 As shown, the threshold vector is calculated based on the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector, including: S310, cascading the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector to obtain an interactive speech cascade feature vector; and, S320, first multiplying the interactive speech cascade feature vector with a transformation matrix, then adding the obtained feature vector and the bias vector by position, and then inputting the obtained feature vector into a sigmo id activation function to obtain the threshold vector. In particular, the range of each threshold value in the threshold vector is greater than or equal to zero and less than or equal to one.

[0054] Figure 5 In the interactive response method based on the large language model according to the embodiment of the present application, the threshold value of each position in the threshold vector is used to calculate the weighted sum of the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector to obtain the interactive text semantic-interactive emotion fusion representation vector. More specifically, in the embodiment of the present application, Figure 5 As shown, with the threshold value of each position in the threshold vector, the weighted sum of the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector is calculated to obtain the interactive text semantic-interactive emotion fusion representation vector, including: S410, using the eigenvalue of each position in the interactive speech text semantic coding feature vector as the exponent of the natural constant to calculate the product between the exponential function value with the natural constant as the base and one minus the corresponding threshold value to obtain an interactive speech text semantic coding feature vector composed of multiple interactive speech text semantic feature values; S420, using the eigenvalue of each position in the interactive speech emotion pattern waveform semantic representation vector as the exponent of the natural constant to calculate the product between the exponential function value with the natural constant as the base and the corresponding threshold value to obtain an interactive speech emotion pattern waveform semantic coding feature vector composed of multiple interactive speech emotion pattern waveform semantic feature values; and, S430, adding the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic coding feature vector by position to obtain the interactive text semantic-interactive emotion fusion representation vector.

[0055] In an embodiment of the present application, specifically, the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector are input into a feature value-by-feature value granularity interactive fusion module to obtain an interactive text semantic-interactive emotion fusion representation vector, including: using the feature value-by-feature value granularity interactive fusion module to fuse the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector using the following fusion formula to obtain the interactive text semantic-interactive emotion fusion representation vector; wherein the fusion formula is:

[0056]

[0057] Wherein, V1 and V2 are respectively the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector, concat(·,·) represents the cascade operation, V is the interactive speech cascade feature vector, v1 and v2 are respectively the feature values ​​of each position in the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector, v c is the eigenvalue of each position in the interactive text semantic-interactive emotion fusion representation vector, T is the threshold vector, t is the threshold value of each position in the threshold vector, t∈[0,1], W is the transformation matrix, b is the bias vector, and sigmoid(·) represents the activation function.

[0058] In step S170, based on the interactive text semantic-interactive emotion fusion representation feature, the digital human interactive text is obtained and played. Specifically, in an embodiment of the present application, based on the interactive text semantic-interactive emotion fusion representation feature, the digital human interactive text is obtained and played, including: inputting the interactive text semantic-interactive emotion fusion representation vector into an interactive content generator based on AIGC to obtain the digital human interactive text; and converting the digital human interactive text into a voice signal and playing it in the interactive window. That is, the interactive text semantic-interactive emotion fusion representation feature obtained by fusing the interactive voice text semantic encoding feature vector and the interactive voice emotion pattern waveform semantic representation vector at a feature value granularity is used to intelligently generate the digital human interactive text and play it in the interactive window. In this way, the user's query intention and context information can be deeply understood, and the user's emotional state can be identified at the same time, and the emotional color of the reply can be adjusted accordingly to make the interaction process more natural, thereby improving the overall user experience.

[0059] Preferably, inputting the interactive text semantic-interactive emotion fusion representation vector into an interactive content generator based on AIGC to obtain a digital human interactive text includes the following steps: calculating the self-mean matrix and the self-standard deviation matrix of the interactive text semantic-interactive emotion fusion representation vector, wherein the value of the (i, j)th position of the self-mean matrix is ​​the mean of the eigenvalue at the i-th position and the eigenvalue at the j-th position of the interactive text semantic-interactive emotion fusion representation vector, and the value of the (i, j)th position of the self-standard deviation matrix is ​​the standard deviation of the eigenvalue at the i-th position and the eigenvalue at the j-th position of the interactive text semantic-interactive emotion fusion representation vector; performing matrix multiplication of the transposed vector of the interactive text semantic-interactive emotion fusion representation vector as a row feature vector with the self-mean matrix to obtain the interactive text semantic-interactive emotion fusion mean mapping vector, and performing matrix multiplication of the self-standard deviation matrix with the interactive text semantic-interactive emotion fusion representation vector to obtain the interactive text semantic-interactive emotion fusion mean mapping vector. The method comprises the following steps: calculating the point sum of the interactive text semantics-interactive emotion fusion mean mapping vector and the transposed vector of the interactive text semantics-interactive emotion fusion standard deviation mapping vector to obtain the interactive text semantics-interactive emotion fusion mean-standard deviation vector; calculating the matrix product of the self-mean matrix and the self-standard deviation matrix, and performing matrix multiplication of the transposed vector of the interactive text semantics-interactive emotion fusion representation vector and the matrix product to obtain the interactive text semantics-interactive emotion fusion aggregate reference representation vector; calculating the point sum of the interactive text semantics-interactive emotion fusion mean-standard deviation vector and the transposed vector of the interactive text semantics-interactive emotion fusion aggregate reference representation vector to obtain the optimized interactive text semantics-interactive emotion fusion representation vector; and inputting the optimized interactive text semantics-interactive emotion fusion representation vector into the AIGC-based interactive content generator to obtain the digital human interactive text.

[0060] In particular, here, considering the interactive fusion representation of the interactive text semantic-interactive emotion fusion representation vector for the feature value granularity of the text semantic encoding features of the interactive speech text and the signal waveform image semantic features of the interactive speech signal, the difference in the correspondence of the feature value granularity of the feature distribution caused by the data modality difference will cause the interactive text semantic-interactive emotion fusion representation vector as a whole to have local overflow characteristics, thereby affecting the mapping accuracy of the interactive text semantic-interactive emotion fusion representation vector to the generated text semantic space when the interactive content generator based on AIGC generates digital human interactive text.

[0061] Based on this, the self-mean matrix and self-standard deviation matrix of the group aggregation statistical evaluation of the local distribution of the feature value granularity of the interactive text semantic-interactive emotion fusion representation vector are used as the retrieval and response distribution enhancement of the interactive text semantic-interactive emotion fusion representation vector, and a reference-free distribution retrieval response framework under the open domain of the feature distribution based on group aggregation of the interactive text semantic-interactive emotion fusion representation vector is constructed to avoid the distribution response redundancy caused by the local overflow characteristics of the interactive text semantic-interactive emotion fusion representation vector through response superposition, so as to realize the faithfulness constraint of the response self-aggregation statistics related to the generated text semantic space domain of the optimized interactive text semantic-interactive emotion fusion representation vector, and improve the text quality of the digital human interactive text obtained by inputting the optimized interactive text semantic-interactive emotion fusion representation vector into the interactive content generator based on AI GC. In this way, the user's query intention and context information can be deeply understood, and the user's emotional state can be identified at the same time, and the emotional color of the reply can be adjusted accordingly to make the interaction process more natural, thereby improving the overall user experience.

[0062] In summary, the interactive response method based on the large language model of the embodiment of the present application is explained, which generates an interactive window in response to detecting the action of the user determining to interact with the first digital human object, and in the interactive window, receives the interactive voice signal input by the user, and uses artificial intelligence-based voice signal processing and recognition technology to perform voice processing and text semantic encoding on the interactive voice signal, so as to intelligently generate digital human interactive text based on the fusion features of the text semantics and emotional waveform semantics of the interactive voice signal and play it in the interactive window. In this way, the user's query intention and context information can be deeply understood, and the user's emotional state can be identified, and the emotional color of the reply can be adjusted accordingly, making the interaction process more natural, thereby improving the overall user experience.

[0063] Figure 6 FIG. 1 is a block diagram of an interactive response system based on a large language model according to an embodiment of the present application. Figure 6As shown, according to the embodiment of the present application, the interactive response system 100 based on the large language model includes: an interactive window generation module 110, which is used to generate an interactive window in response to detecting an action of a user determining to interact with a first digital human object; an interactive voice signal receiving module 120, which is used to receive an interactive voice signal input by the user in the interactive window; an interactive voice signal recognition module 130, which is used to perform voice recognition on the interactive voice signal to obtain an interactive voice text recognition result; an interactive voice text semantic encoding module 140, which is used to perform semantic encoding on the interactive voice text recognition result to obtain an interactive voice text semantic encoding feature. vector; an emotion pattern feature extraction module 150, which is used to extract the emotion pattern features of the interactive voice signal to obtain an interactive voice emotion pattern waveform semantic representation vector; a semantic emotion fusion module 160, which is used to input the interactive voice text semantic encoding feature vector and the interactive voice emotion pattern waveform semantic representation vector into a feature value-by-feature value granularity interactive fusion module to obtain an interactive text semantic-interactive emotion fusion representation vector as an interactive text semantic-interactive emotion fusion representation feature; and a digital human interactive text generation module 170, which is used to obtain and play the digital human interactive text based on the interactive text semantic-interactive emotion fusion representation feature.

[0064] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned interactive response system based on the large language model have been described in the above reference. Figures 1 to 5 The description of the interactive response method based on a large language model has been introduced in detail, and therefore, its repeated description will be omitted.

[0065] As described above, the interactive response system 100 based on the large language model according to the embodiment of the present disclosure can be implemented in various wireless terminals, such as a server with an interactive response algorithm based on the large language model. In a possible implementation, the interactive response system 100 based on the large language model according to the embodiment of the present disclosure can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the interactive response system 100 based on the large language model can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the interactive response system 100 based on the large language model can also be one of the many hardware modules of the wireless terminal.

[0066] Alternatively, in another example, the large language model-based interactive response system 100 and the wireless terminal may be separate devices, and the large language model-based interactive response system 100 may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0067] The foregoing is merely an example of the principles of the present disclosure, and various modifications may be made by those skilled in the art without departing from the scope of the present disclosure. The above embodiments are presented for the purpose of illustration and not limitation. The present disclosure may also take many forms other than those explicitly described herein. Therefore, it is emphasized that the present disclosure is not limited to the methods, systems and devices explicitly disclosed, but is intended to include variations and modifications within the spirit and scope of the appended claims.

Claims

1. An interactive response method based on a large language model, characterized in that: include: In response to detecting an action of the user determining to interact with the first digital human object, generating an interaction window; In the interactive window, receiving an interactive voice signal input by the user; Performing speech recognition on the interactive speech signal to obtain an interactive speech text recognition result; Performing semantic coding on the interactive speech text recognition result to obtain an interactive speech text semantic coding feature vector; Extracting emotion pattern features from the interactive speech signal to obtain an interactive speech emotion pattern waveform semantic representation vector; Input the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector into a feature value-by-feature value granularity interactive fusion module to obtain an interactive text semantic-interactive emotion fusion representation vector as an interactive text semantic-interactive emotion fusion representation feature; Based on the interactive text semantics-interactive emotion fusion representation feature, a digital human interactive text is obtained and played; The interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector are input into the feature value-by-feature value granularity interactive fusion module to obtain the interactive text semantic-interactive emotion fusion representation vector as the interactive text semantic-interactive emotion fusion representation feature, including: Calculate a threshold vector based on the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector; Using the threshold values ​​at each position in the threshold vector, a weighted sum of the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector is calculated to obtain the interactive text semantic-interactive emotion fusion representation vector; The step of calculating the threshold vector based on the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector comprises: Cascading the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector to obtain an interactive speech cascade feature vector; The interactive speech cascade feature vector is first multiplied by the transformation matrix, the obtained feature vector is then added to the bias vector by position, and then the obtained feature vector is input into a sigmoid activation function to obtain the threshold vector; Among them, the weighted sum of the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector is calculated based on the threshold value of each position in the threshold vector to obtain the interactive text semantic-interactive emotion fusion representation vector, including: Taking the feature value of each position in the interactive speech text semantic coding feature vector as the exponent of the natural constant to calculate the product between the exponential function value with the natural constant as the base and one minus the corresponding threshold value to obtain the interactive speech text semantic coding feature vector composed of multiple interactive speech text semantic feature values; Taking the eigenvalues ​​of each position in the interactive speech emotion pattern waveform semantic representation vector as the exponent of the natural constant to calculate the product between the exponential function value with the natural constant as the base and the corresponding threshold value to obtain an interactive speech emotion pattern waveform semantic coding feature vector composed of multiple interactive speech emotion pattern waveform semantic eigenvalues; The interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic coding feature vector are added according to position to obtain the interactive text semantic-interactive emotion fusion representation vector.

2. The interactive response method based on a large language model according to claim 1, characterized in that: The interactive speech text recognition result is semantically encoded to obtain an interactive speech text semantic encoding feature vector, including: performing word segmentation processing on the interactive speech text recognition result and then passing it through an interactive speech text semantic encoder based on a BERT model to obtain the interactive speech text semantic encoding feature vector.

3. The interactive response method based on a large language model according to claim 2, characterized in that: The interactive speech signal is subjected to emotion pattern feature extraction to obtain an interactive speech emotion pattern waveform semantic representation vector, comprising: inputting the interactive speech signal into an emotion pattern feature extractor based on a convolutional neural network model to obtain the interactive speech emotion pattern waveform semantic representation vector.

4. The interactive response method based on a large language model according to claim 3, characterized in that: The range of each threshold value in the threshold vector is greater than or equal to zero and less than or equal to one.

5. The interactive response method based on a large language model according to claim 4, characterized in that: Based on the interactive text semantics-interactive emotion fusion representation feature, the digital human interactive text is obtained and played, including: Inputting the interactive text semantic-interactive emotion fusion representation vector into an interactive content generator based on AIGC to obtain a digital human interactive text; The digital human interaction text is converted into a voice signal and played in the interaction window.

6. An interactive response system based on a large language model, characterized in that: include: An interaction window generating module, configured to generate an interaction window in response to detecting an action of a user determining to interact with the first digital human object; An interactive voice signal receiving module, used for receiving an interactive voice signal input by the user in the interactive window; An interactive voice signal recognition module, used to perform voice recognition on the interactive voice signal to obtain an interactive voice text recognition result; An interactive speech text semantic coding module, used for semantically coding the interactive speech text recognition result to obtain an interactive speech text semantic coding feature vector; An emotion pattern feature extraction module is used to extract emotion pattern features from the interactive speech signal to obtain an interactive speech emotion pattern waveform semantic representation vector; A semantic emotion fusion module, used for inputting the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector into a feature value-by-feature value granularity interactive fusion module to obtain an interactive text semantic-interactive emotion fusion representation vector as an interactive text semantic-interactive emotion fusion representation feature; A digital human interactive text generation module is used to obtain and play the digital human interactive text based on the interactive text semantics-interactive emotion fusion representation feature; The interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector are input into the feature value-by-feature value granularity interactive fusion module to obtain the interactive text semantic-interactive emotion fusion representation vector as the interactive text semantic-interactive emotion fusion representation feature, including: Calculate a threshold vector based on the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector; Using the threshold values ​​at each position in the threshold vector, a weighted sum of the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector is calculated to obtain the interactive text semantic-interactive emotion fusion representation vector; The step of calculating the threshold vector based on the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector comprises: Cascading the interactive speech text semantic encoding feature vector and the interactive speech emotion pattern waveform semantic representation vector to obtain an interactive speech cascade feature vector; The interactive speech cascade feature vector is first multiplied by the transformation matrix, the obtained feature vector is then added to the bias vector by position, and then the obtained feature vector is input into a sigmoid activation function to obtain the threshold vector; Among them, the weighted sum of the interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic representation vector is calculated based on the threshold value of each position in the threshold vector to obtain the interactive text semantic-interactive emotion fusion representation vector, including: Taking the feature value of each position in the interactive speech text semantic coding feature vector as the exponent of the natural constant to calculate the product between the exponential function value with the natural constant as the base and one minus the corresponding threshold value to obtain the interactive speech text semantic coding feature vector composed of multiple interactive speech text semantic feature values; Taking the eigenvalues ​​of each position in the interactive speech emotion pattern waveform semantic representation vector as the exponent of the natural constant to calculate the product between the exponential function value with the natural constant as the base and the corresponding threshold value to obtain an interactive speech emotion pattern waveform semantic coding feature vector composed of multiple interactive speech emotion pattern waveform semantic eigenvalues; The interactive speech text semantic coding feature vector and the interactive speech emotion pattern waveform semantic coding feature vector are added according to position to obtain the interactive text semantic-interactive emotion fusion representation vector.

7. The interactive response system based on a large language model according to claim 6, characterized in that: The emotion pattern feature extraction module is used to: input the interactive speech signal into an emotion pattern feature extractor based on a convolutional neural network model to obtain the interactive speech emotion pattern waveform semantic representation vector.

Citation Information

Patent Citations

  • Multi-mode question and answer method and system based on emotion recognition and large language model

    CN117909486A