Multimodal digital human interaction method and system based on large language model

Abstract

This application relates to the technical field of large language models, and specifically to a multimodal digital human interaction method and system based on a large language model. The method includes: obtaining video data, audio signals, and text data of a single conversation when a user interacts with a digital human; extracting the envelope of the audio signal to determine the emotion evaluation value and semantic contrast of the single conversation; identifying the emotions contained in each frame of the video data to form an emotion sequence for the single conversation; obtaining the semantic deviation of the single conversation; obtaining the semantic guidance value of the single conversation; and combining with an emotional conversation generation model to generate a corresponding emotional conversation. This application can reduce the deviation in the understanding of the user's expressed intentions and the perception of emotions, more accurately judge the user's emotional state, so that the generated emotional response content matches the user's emotional state, and improve the user's interactive experience.

Description

Technical Field

[0001] The present application relates to the technical field of large language models, and in particular to a multimodal digital human interaction method and system based on a large language model. Background Art

[0002] Due to the high volume of customer inquiries and the repetitive nature of the content, traditional manual customer service practices are characterized by high workload, low service efficiency, and unstable service quality. The introduction of digital human technology has alleviated this problem, but traditional human-digital human interaction scenarios typically employ a single, passive response model, lacking the ability to actively perceive and dynamically understand users' emotional states. This results in a mechanical interaction process and a stilted user experience.

[0003] Due to the ambiguity of text characters in the natural language processing process and the multidimensional expression of human emotions, such as tone and speaking speed, this single-modal sentiment analysis method will have deviations in its understanding of human intentions and perception of emotions when faced with the ambiguity of natural language and the complexity of human emotional expression. It cannot accurately judge human emotions and intentions, resulting in a mismatch between the emotional response content generated by the digital human and the user's emotional state, making the interaction process lack emotional resonance and the user interaction experience poor. Summary of the Invention

[0004] In order to solve the above technical problems, a multimodal digital human interaction method and system based on a large language model are provided to solve the existing problems.

[0005] The solution to the technical problem of this application is to provide a multimodal digital human interaction method and system based on a large language model, including the following steps:

[0006] In a first aspect, an embodiment of the present application provides a multimodal digital human interaction method based on a large language model, the method comprising the following steps:

[0007] Obtain video data, audio signals, and text data of a single conversation between a user and a digital human;

[0008] Extracting the envelope of the audio signal, analyzing the extreme range and average level of intervals between adjacent peaks on the envelope, and determining the emotional assessment value of a single conversation based on the discrete conditions of all peak values;

[0009] Identifying polysemous words, reduplications, and antonyms contained in the text data; calculating the semantic contrast of a single conversation based on the proportion of the polysemous words, reduplications, and antonyms in the text data;

[0010] Identifying the emotions contained in each frame of the video data to form an emotion sequence of a single conversation; analyzing the discreteness of elements in the emotion sequence, and combining the semantic contrast with the emotion evaluation value to obtain a semantic deviation amount of the single conversation;

[0011] The semantic guidance value of a single conversation is obtained by combining the deviation of the largest element in the emotional sequence of a single conversation and the difference in the emotional sequences between two adjacent conversations with the semantic deviation.

[0012] Based on the semantic guidance value and the text data, in combination with an emotional dialogue generation model, a corresponding emotional dialogue is generated.

[0013] Preferably, an envelope extraction algorithm is used to extract the envelope of the audio signal.

[0014] Preferably, determining the emotion evaluation value of a single conversation includes:

[0015] Obtain all peaks on the envelope;

[0016] Calculate the time interval between the moments corresponding to two adjacent peaks on the envelope; calculate the ratio of the range and the mean of the time intervals between all two adjacent peaks on the envelope, and record it as the relative ratio;

[0017] Calculate the degree of dispersion of all peak values on the envelope, which is recorded as the first dispersion;

[0018] The emotion evaluation value is a normalized result of the sum of the relative ratio and the first dispersion.

[0019] Preferably, the identifying of polysemous words, reduplications and antonyms contained in the text data includes: using a polysemous word library and an antonym pair library and employing regular expressions to identify the polysemous words, reduplications and antonyms contained in the text data.

[0020] Preferably, the calculation of the semantic contrast of a single conversation includes taking the ratio of the sum of the total number of words of all polysemous words, the total number of words of all reduplications, and the total number of words of all antonyms in the text data to the total number of words of all words in the text data as the semantic contrast of the single conversation.

[0021] Preferably, the emotion sequence is further acquired by performing frame processing on the video data to acquire each frame image, identifying the emotion label corresponding to each frame image through a neural network algorithm, and forming the emotion labels of all frame images corresponding to a single conversation into an emotion sequence.

[0022] Preferably, obtaining the semantic deviation amount of a single conversation includes:

[0023] Calculating the discrete degree of the emotion sequence, recorded as the second discrete degree;

[0024] The accumulated sum of the semantic contrast and the emotion evaluation value is calculated, and a normalized result of the product of the accumulated sum and the second dispersion is used as the semantic deviation amount of the single conversation.

[0025] Preferably, obtaining the semantic guidance value of a single conversation includes:

[0026] Calculate the difference between the mode of the emotion sequence in a single conversation and the mode of the emotion sequence in the previous conversation, and record it as the emotion difference;

[0027] The difference between the maximum value of the emotion sequence in a single conversation and a preset value is recorded as the emotion deviation;

[0028] The sum of the emotion deviation and the emotion difference is calculated, and a normalized value of the product of the sum and the semantic deviation is used as the semantic guidance value of a single conversation.

[0029] Preferably, generating the corresponding emotional dialogue includes:

[0030] In the back propagation process of the emotional dialogue generation model, random variables are introduced when updating the weights, so that the weight update formula is: ,in, is the updated weight, is the weight before updating, is the learning rate, is the gradient calculated by the loss function, is the semantic guide value, is the introduced random variable, where Obey normal distribution;

[0031] The text data and the semantic guidance value are used as inputs of the emotional dialogue generation model, and in the back-propagation process, random variables are introduced to update the weights based on the semantic guidance value, and the corresponding emotional dialogue is generated in combination with the updated weights.

[0032] In the second aspect, an embodiment of the present application also provides a multimodal digital human interaction system based on a large language model, comprising a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the above-mentioned multimodal digital human interaction methods based on a large language model.

[0033] This application has at least the following beneficial effects:

[0034] The present application obtains multiple modal data, including video, text and audio data, to perceive the user's emotions from multiple dimensions, so as to grasp the user's emotions more accurately; by analyzing the time intervals between adjacent peaks on the envelope line in the audio signal, the emotional evaluation value of a single conversation is calculated, which has the beneficial effect of taking into account the urgency of the user's speaking speed and tone during the expression process, so as to reflect the user's emotional fluctuations; secondly, by calculating the semantic contrast of a single conversation through the presence of polysemous words, antonyms and reduplications in the text data, the beneficial effect is that it takes into account the complexity of the context in the user's expression sentences, as well as the irony in the context, to further reflect the user's emotional state; by combining the changes in the user's facial expressions in the video data with the semantic contrast and the emotional evaluation value, the semantic deviation of a single conversation is obtained, which has the beneficial effect of taking into account the user's facial expressions when expressing, and then perceiving the user's emotional state, and by fusing the results of emotional perception of multiple modal data, the user's emotions and the ambiguity of semantic expression are explained to reflect The method reflects the possible deviation in the understanding of user semantics, and obtains the semantic guidance value of a single conversation through the difference in the changes in the user's facial expressions in two adjacent conversations. Its beneficial effect is that it analyzes the change in the user's emotions caused by dissatisfaction with the robot's expression content in the two previous and subsequent conversations, so as to reflect that the reply content generated by the robot at this time should match the user's emotions; based on the semantic guidance value and the text data, combined with the emotional dialogue generation model, the corresponding emotional dialogue is generated. Its beneficial effect is that by adding a random variable based on the semantic guidance value to update the weight during the back propagation of the model, the weight has a certain degree of randomness in the updating process, so as to explore a larger solution space and improve the model's emotional dialogue under the user's complex text requirements. Therefore, through the emotional analysis of multiple modalities, the deviation in the understanding of the user's intentions and the perception of emotions can be reduced, and the user's emotional state can be more accurately judged, so that the emotional reply content generated by the model matches the user's emotional state, thereby improving the user's interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The multimodal digital human interaction method based on a large language model of the present application is further described in detail below with reference to the accompanying drawings.

[0036] Figure 1 A flowchart of the steps of the multimodal digital human interaction method based on a large language model provided in an embodiment of the present application;

[0037] Figure 2 A flowchart of the steps of the method for obtaining semantic guidance values provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] To make the objectives, technical solutions, and advantages of this application more clearly understood, the following, in conjunction with the accompanying drawings and implementation examples, further describes in detail the multimodal digital human interaction method and system based on a large language model proposed in this application. It should be understood that the specific embodiments described herein are merely intended to explain this application and are not intended to limit this application.

[0039] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0040] See also Figure 1 , which shows a flowchart of a multimodal digital human interaction method based on a large language model provided by an embodiment of the present application, the method comprising the following steps:

[0041] Step 1: Obtain video data, audio signals, and text data of a single conversation when the user interacts with the digital human.

[0042] The process of information exchange between humans and digital humans primarily involves engaging in emotional dialogue. Emotional dialogue tasks can be divided into two subtasks: conversation emotion perception and emotional dialogue generation. The former focuses on classifying emotions in the conversation, while the latter focuses on generating emotional responses. Emotional dialogue generation technologies include pure text dialogue generation and multimodal dialogue generation. Pure text dialogue generation analyzes and processes text data to generate responses. However, due to the polysemous nature of text, emotional perception of text information can lead to semantic ambiguity, resulting in inaccurate semantic perception and affecting the quality of emotional dialogue generation. Multimodal dialogue generation improves the quality of conversations between humans and digital humans by comprehensively processing multimodal data such as audio, video, and text to perform emotional perception of the conversation.

[0043] Based on the above analysis, a digital human is deployed on a robot. The robot's camera and microphone array collect video data and audio signals from a single user's interaction with the robot. The audio signal is filtered to remove environmental noise through a filtering algorithm and then converted into text data. The process from the user asking a question to the robot responding is considered a single conversation, from which the video data, audio signal, and text data of the single conversation are obtained.

[0044] In this embodiment, the Kalman filter algorithm is used to filter the audio signal. The Kalman filter algorithm and the audio-to-text conversion process are well-known technologies and will not be described in detail here.

[0045] At this point, the video data, audio signal, and text data of a single conversation are obtained.

[0046] Step 2: extract the envelope of the audio signal, analyze the extreme range and average level of the interval time between adjacent peaks on the envelope, and determine the emotion evaluation value of a single conversation based on the discrete conditions of all peak values.

[0047] During human-computer interaction, the user's intention can be identified through the user's audio signal and text data. The audio signal can reflect the user's emotional level, while the text data can more intuitively express the user's specific intention. Among them, the amplitude and speed of the user's voice in the audio signal will generate a corresponding electrical signal, which can express the user's emotional state. After the user expresses the corresponding intention in a relatively calm tone, if the robot's answer is not enough to solve the user's problem, the user will have an emotional expression, which may cause the expression speed to increase, the intonation between each audio, the fluctuation of the audio symbols is large, and there may be large fluctuations in the audio amplitude. Therefore, by analyzing the changes in the signal amplitude in the audio signal, the emotion evaluation value is calculated to evaluate the user's emotion, specifically:

[0048] Using an envelope extraction algorithm to extract the envelope of the audio signal;

[0049] In this embodiment, the Hilbert transform algorithm is used to extract the envelope, wherein the Hilbert transform algorithm is a well-known technology and will not be described in detail here. As other implementation methods, the implementer can adopt other methods of the existing technology, such as the cepstrum method, etc. This embodiment does not impose any special restrictions on this.

[0050] Obtain all peaks on the envelope;

[0051] In this embodiment, the AMPD (Automatic multiscale-based peak detection) algorithm is used to obtain the peak value. The AMPD algorithm is a well-known technology and will not be described in detail here.

[0052] Calculate the time interval between the moments corresponding to adjacent peaks on the envelope;

[0053] Calculate the ratio of the range and the mean of the time intervals of all adjacent peaks on the envelope, and record it as a relative ratio;

[0054] Calculate the degree of dispersion of all peak values on the envelope, which is recorded as the first dispersion;

[0055] In this embodiment, the degree of dispersion is measured by calculating the variance of all peak values on the envelope. As other implementation methods, the implementer may adopt other methods of the prior art, such as standard deviation, etc. This embodiment does not impose any special restrictions on this.

[0056] Normalizing the sum of the relative ratio and the first dispersion as the emotion evaluation value of the single conversation;

[0057] In this embodiment, the sigmoid function is used for normalization processing, wherein the sigmoid function is a well-known technology and will not be described in detail here. As other implementation methods, the implementer can adopt other methods of the existing technology, such as the softmax function, etc. This embodiment does not impose any special restrictions on this.

[0058] It should be noted that the time interval reflects the time interval between adjacent syllables during the user's language expression. The larger the range, the uneven the speaking speed of the user during the language expression, and the ups and downs between different syllable segments. The smaller the range, the relatively stable time interval between adjacent peaks, and the relatively stable speaking speed of the user during the entire conversation. The smaller the mean, the faster the speaking speed of the user during the language expression, and the urgent nature of the expression. The larger the first discreteness, the more drastic the change in the strength of the voice signal, and the larger the obtained emotion evaluation value, reflecting that the user's emotional fluctuations are more significant.

[0059] At this point, the emotion evaluation value of a single conversation is obtained.

[0060] Step 3: Identify polysemous words, reduplication words, and antonyms contained in the text data; calculate the semantic contrast of a single conversation based on the proportion of the number of polysemous words, reduplication words, and antonyms in the text data.

[0061] Secondly, the user's expressed intent needs to be reflected through text. Text, composed of words or characters, contains emotional vocabulary and semantic information, and is an important vehicle for emotional expression. However, due to the multidimensional nature of human emotions and the complex nature of text, the same word can have different meanings in different contexts. When human emotions are expressed through text, they often include sarcasm and polysemy. Therefore, when users express sarcasm or ridicule in their speech, they often use reduplication or include antonyms in sentences, such as "Right, right, right" and "You're so smart, you messed up such a simple thing." These statements convey the opposite meaning of the word. Furthermore, polysemous words can appear in the text data of a single conversation, expressing ideas that contradict the stated meaning.

[0062] Based on the above analysis, the semantic contrast is calculated by analyzing the characteristics of word meanings in text data, specifically:

[0063] Based on the polysemous word library and the antonym pair library, regular expressions are used to identify polysemous words, reduplications, and antonyms contained in the text data;

[0064] It should be noted that the polysemous word library and the antonym pair library can be constructed by collecting and organizing a large number of polysemous words and antonym pairs, or by using public polysemous word dictionaries and antonym pair dictionaries as polysemous word libraries and antonym pair libraries, and using regular expressions to search for antonyms, polysemous words and reduplications appearing in text data. Among them, regular expressions are a well-known technology and will not be described in detail here.

[0065] The ratio of the sum of the total number of words in all polysemous words, the total number of words in all reduplication words, and the total number of words in all antonyms in the text data to the total number of words in the text data is used as the semantic contrast of a single conversation;

[0066] It should be noted that since Chinese characters pay more attention to context, when users express irony and sarcasm in the form of statements to show their dissatisfaction with the robot's answer, more sarcastic text features often appear, such as certain reduplications to emphasize attitudes, more antonyms to reflect irony, and more synonyms to construct complex contexts. Therefore, these three types of words account for a large proportion in the text data, and the resulting semantic contrast is high, indicating that the text data in this conversation contains more complex semantics and the user's expression uses more sarcastic meanings, which in turn reflects the user's emotional state.

[0067] At this point, the semantic contrast of a single conversation is obtained.

[0068] Step 4: Identify the emotions contained in each frame of the video data to form an emotion sequence of a single conversation; analyze the discreteness of the elements in the emotion sequence, and combine the semantic contrast with the emotion evaluation value to obtain the semantic deviation of the single conversation.

[0069] During a single conversation, when a user expresses their intention, their facial expressions will contain emotional information. To understand the user's emotional state through the user's facial expressions in the video data, it is necessary to recognize the user's facial expressions in the video data. Specifically:

[0070] Performing frame processing on the video data to obtain each frame image, identifying each frame image using a neural network algorithm, obtaining an emotion label corresponding to each frame image, and forming an emotion sequence from the emotion labels of all frame images corresponding to a single conversation;

[0071] In this embodiment, the ResNet18 algorithm is used to identify the emotion of each frame of image and obtain the emotion label of each frame of image. The ResNet18 algorithm is trained by using the public data set FER2013, wherein 7 emotions are marked in the data set, including anger, disgust, sadness, fear, neutrality, surprise and happiness. Each emotion corresponds to an emotion label, specifically: anger corresponds to emotion label 1, disgust corresponds to emotion label 2, sadness corresponds to emotion label 3, fear corresponds to emotion label 4, neutrality corresponds to emotion label 5, surprise corresponds to emotion label 6 and happiness corresponds to emotion label 7. Therefore, the emotion of each frame of image is identified by the trained ResNet18 algorithm, and the emotion label of each frame of image is obtained. The ResNet18 algorithm is a well-known technology and will not be described in detail here.

[0072] Secondly, the semantic deviation is calculated by considering the user's facial expression, the semantic situation of the text, and the emotional state in the audio. Specifically:

[0073] Calculating the discrete degree of the emotion sequence, recorded as the second discrete degree;

[0074] In this embodiment, the degree of dispersion is measured by calculating the standard deviation of the emotion sequence. As other implementations, implementers can adopt other methods in the prior art, such as variance, coefficient of variation, etc. This embodiment does not impose any special restrictions on this.

[0075] Calculating a cumulative sum of the semantic contrast and the emotion evaluation value, and normalizing a product of the cumulative sum and the second dispersion as a semantic deviation amount of a single conversation;

[0076] In this embodiment, the sigmoid function is used for normalization processing, wherein the sigmoid function is a well-known technology and will not be described in detail here. As other implementation methods, the implementer can adopt other methods of the existing technology, such as the softmax function, the tanh function, etc., and this embodiment does not impose any special restrictions on this.

[0077] It should be noted that when the content expressed by the robot is not sufficient to solve the user's actual problem, or when the user is anxious when asking questions, it will affect the user's language expression, which may cause changes in the user's emotions, resulting in the user being affected by the multidimensionality of emotions and the complexity of Chinese characters in multimodal data, making the expression of word meanings contrary to the true intention. Correspondingly, the greater the fluctuation of the emotion label of each frame image in the video data, the greater the second discreteness obtained; the faster the speech speed in the audio data and the greater the fluctuation, the greater the emotion evaluation value obtained; the higher the semantic complexity in the text data, the greater the semantic contrast obtained. Therefore, the larger the semantic deviation obtained, indicating that the user's emotions are intense and the semantic expression is obscure, and the deviation of the robot's understanding of the user's semantics may be greater.

[0078] Step 5: The semantic guidance value of a single conversation is obtained by combining the deviation of the largest element in the emotion sequence of a single conversation and the difference in the emotion sequences between two adjacent conversations with the semantic deviation; based on the semantic guidance value and the text data, the corresponding emotion conversation is generated in combination with the emotion conversation generation model.

[0079] Furthermore, due to the influence of the two previous conversations, the robot's expression in a certain conversation may not satisfy the user, or the answer content may be poor, resulting in a change in the user's mood. Therefore, the semantic guidance value is determined by combining the emotional changes of the user's facial expressions in two adjacent conversations with the semantic deviation amount, specifically:

[0080] Calculate the difference between the mode of the emotion sequence in a single conversation and the mode of the emotion sequence in the previous conversation, and record it as the emotion difference;

[0081] In this embodiment, the absolute value of the difference between the mode of the emotion sequence of a single conversation and the mode of the emotion sequence of the previous conversation is calculated and recorded as the emotion difference.

[0082] It should be noted that if this is the first time that the user and the robot are talking to each other, the value of the emotion difference is 0.

[0083] The difference between the maximum value of the emotion sequence in a single conversation and a preset value is recorded as the emotion deviation;

[0084] In this embodiment, since the emotion label corresponding to the happy emotion in the facial expression is 7, the preset value is set to 7. As for other implementation methods, the implementer can set it according to the actual situation; therefore, the absolute value of the difference between the maximum value in the emotion sequence of a single conversation and the preset value is recorded as the emotion deviation.

[0085] Calculating the sum of the emotion deviation and the emotion difference, and taking the normalized value of the product of the sum and the semantic deviation as the semantic guidance value of the single conversation;

[0086] In this embodiment, the calculation formula for the semantic guidance value of a single conversation is:

[0087]

[0088] in, is the semantic guidance value of a single conversation, is the semantic deviation of a single conversation, is the emotional difference of a single conversation, is the emotional deviation of a single conversation, To avoid The result is 0. In this embodiment, a value greater than 0 is preset. Set to 1. As other implementation methods, implementers can set it according to actual conditions. It is a normalization function. In this embodiment, the sigmoid function is used for normalization. The sigmoid function is a well-known technology and will not be described in detail here. As other implementation methods, the implementer can adopt other methods of the existing technology, such as the softmax function, the tanh function, etc. This embodiment does not impose any special restrictions on this.

[0089] It should be noted that, during the two previous conversations, the greater the user's emotional fluctuations, that is, the greater the emotional difference, and the greater the deviation between the emotional labels corresponding to the happy labels, that is, the greater the emotional deviation, the content expressed by the robot at this time cannot be recognized by the customer, and it may affect the robot's logical analysis of the user's semantics with more complex expressions, which is more likely to cause semantic ambiguity. The larger the obtained semantic guidance value, the more the robot should generate a dialogue with soothing emotions. Among them, the step flow chart of the method for obtaining the semantic guidance value provided in the embodiment of the present application is as follows Figure 2 shown.

[0090] The emotional dialogue generation model is trained. In this embodiment, the variational autoencoder model (VAE) is trained. Therefore, the training process is as follows:

[0091] By collecting multimodal data of multiple conversations between multiple users and the robot, the multimodal data includes video data, audio data and text data to form an experimental dataset; calculating the semantic guidance value of each conversation in the experimental dataset;

[0092] In this embodiment, multimodal data of 2,000 users' conversations are collected to form an experimental data set. As other implementation methods, implementers can set them according to actual conditions.

[0093] In the back propagation process of the traditional VAE model, the weight update formula is: , and in this embodiment, the VAE model introduces random variables when updating weights during the back propagation process. Based on the semantic guidance value, the weight update formula is: ,in, is the updated weight, is the weight before updating, is the learning rate, is the gradient calculated by the loss function, is the semantic guide value, is the introduced random variable, where Obeys normal distribution.

[0094] In this embodiment, the learning rate is set to 0.01; wherein, It obeys the normal distribution with a mean of 0 and a standard deviation of the gradient. As other implementation methods, the implementer can set it according to actual conditions.

[0095] Therefore, the VAE model is trained using all the text data in the experimental dataset and the corresponding semantic guidance values;

[0096] The text data and semantic guidance values of a single conversation are used as input to the trained VAE model. During the backpropagation process, random variables are introduced to update the weights based on the semantic guidance values. The updated weights are then combined to generate the corresponding emotional conversation.

[0097] In this embodiment, the VAE model is a well-known technology and will not be described in detail here.

[0098] It should be noted that when training the VAE model, multimodal data of multiple conversations between multiple users and the robot are collected, and the multimodal data includes video data, audio data and text data to form an experimental data set; the semantic guidance value of each conversation in the experimental data set is calculated, and thus the VAE model adds random variables based on the semantic guidance value to update the weights during the back propagation process, so that the weights have a certain degree of randomness during the update process, so as to explore a larger solution space, improve the model's emotional dialogue under the user's complex text requirements, and improve the quality of emotional dialogue. Therefore, the VAE model is trained with the experimental data set and deployed on the robot after training. The robot collects multimodal data of each conversation with the user in real time, and then generates corresponding emotional dialogue according to customer needs and emotional conditions, thereby improving the accuracy of user intention recognition and improving user service quality and efficiency.

[0099] Based on the same inventive concept as the above method, an embodiment of the present application also provides a multimodal digital human interaction system based on a large language model, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the above-mentioned multimodal digital human interaction methods based on a large language model.

[0100] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0101] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the present application. It should be noted that a person skilled in the art can make various modifications and improvements without departing from the spirit of the present application. Therefore, any simple modifications, equivalent variations, and modifications to the above embodiments made in accordance with the technical essence of the present application without departing from the content of the present application's technical solution fall within the scope of protection of the present application's technical solution.

Claims

1. A multimodal digital human interaction method based on a large language model, characterized by: The method comprises the following steps: Obtain video data, audio signals, and text data of a single conversation between a user and a digital human; Extracting the envelope of the audio signal, analyzing the extreme range and average level of intervals between adjacent peaks on the envelope, and determining the emotional assessment value of a single conversation based on the discrete conditions of all peak values; Identifying polysemous words, reduplications, and antonyms contained in the text data; calculating the semantic contrast of a single conversation based on the proportion of the polysemous words, reduplications, and antonyms in the text data; Identifying the emotions contained in each frame of the video data to form an emotion sequence of a single conversation; analyzing the discreteness of elements in the emotion sequence, and combining the semantic contrast with the emotion evaluation value to obtain a semantic deviation amount of the single conversation; The semantic guidance value of a single conversation is obtained by combining the deviation of the largest element in the emotional sequence of a single conversation and the difference in the emotional sequences between two adjacent conversations with the semantic deviation. Based on the semantic guidance value and the text data, combined with the emotional dialogue generation model, a corresponding emotional dialogue is generated; Obtaining the semantic deviation of a single conversation includes: Calculating the discrete degree of the emotion sequence, recorded as the second discrete degree; Calculating a cumulative sum of the semantic contrast and the emotion evaluation value, and normalizing a product of the cumulative sum and the second dispersion as a semantic deviation amount of a single conversation; Obtaining the semantic guidance value of a single conversation includes: Calculate the difference between the mode of the emotion sequence in a single conversation and the mode of the emotion sequence in the previous conversation, and record it as the emotion difference; The difference between the maximum value of the emotion sequence in a single conversation and a preset value is recorded as the emotion deviation; The sum of the emotion deviation and the emotion difference is calculated, and a normalized value of the product of the sum and the semantic deviation is used as the semantic guidance value of a single conversation.

2. The multimodal digital human interaction method based on a large language model according to claim 1, characterized in that: An envelope extraction algorithm is used to extract the envelope of the audio signal.

3. The multimodal digital human interaction method based on a large language model according to claim 1, characterized in that: Determining the emotion evaluation value of a single conversation includes: Obtain all peaks on the envelope; Calculate the time interval between the moments corresponding to two adjacent peaks on the envelope; calculate the ratio of the range and the mean of the time intervals between all two adjacent peaks on the envelope, and record it as the relative ratio; Calculate the degree of dispersion of all peak values on the envelope, which is recorded as the first dispersion; The emotion evaluation value is a normalized result of the sum of the relative ratio and the first dispersion.

4. The multimodal digital human interaction method based on a large language model according to claim 1, characterized in that: The identifying of polysemous words, reduplications and antonyms contained in the text data includes: using a polysemous word library and an antonym pair library and adopting regular expressions to identify the polysemous words, reduplications and antonyms contained in the text data.

5. The multimodal digital human interaction method based on a large language model according to claim 1, characterized in that: The calculation of the semantic contrast of a single conversation includes: taking the ratio of the sum of the total number of words of all polysemous words, the total number of words of all reduplication words, and the total number of words of all antonyms in the text data to the total number of words of all words in the text data as the semantic contrast of the single conversation.

6. The multimodal digital human interaction method based on a large language model according to claim 1, characterized in that: The emotion sequence is further obtained by performing frame processing on the video data to obtain each frame image, identifying the emotion label corresponding to each frame image through a neural network algorithm, and forming an emotion sequence from the emotion labels of all frame images corresponding to a single conversation.

7. The multimodal digital human interaction method based on a large language model according to claim 1, characterized in that: Generating the corresponding emotional dialogue includes: In the back propagation process of the emotional dialogue generation model, random variables are introduced when updating the weights, so that the weight update formula is: ,in, is the updated weight, is the weight before updating, is the learning rate, is the gradient calculated by the loss function, is the semantic guide value, is the introduced random variable, where Obey normal distribution; The text data and the semantic guidance value are used as inputs of the emotional dialogue generation model, and in the back-propagation process, random variables are introduced to update the weights based on the semantic guidance value, and the corresponding emotional dialogue is generated in combination with the updated weights.

8. A multimodal digital human interaction system based on a large language model, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the multimodal digital human interaction method based on a large language model as described in any one of claims 1 to 7 are implemented.

Images