Digital NPC role emotion expression and interaction system

Abstract

The invention discloses a digital NPC role emotion expression and interaction system, relates to the technical field of digital interaction, and aims to improve the accuracy and dimension coverage of user emotion state recognition, comprehensively capture dominant emotion expression and implicit emotion tendency of a user, form more accurate and fine emotion state features and improve the user emotion state recognition efficiency. According to the method and the system, the NPC role response adaptation degree and the user interaction experience satisfaction degree can be improved, the user experience can be improved, the user experience can be improved, and the user experience can be further improved by searching and matching each allowable response emotion type and comprehensively evaluating to obtain a target response emotion type. And meanwhile, the voice output and the body movement of the NPC role present a natural and smooth rhythm sensation, the emotion change of the user can be continuously monitored in the interaction process, and the speech speed, the volume and the movement amplitude parameters of the next interaction segment are dynamically adjusted by calculating the emotion change trend deviation value, so that the real-time adaptive optimization of the NPC role interaction expression is realized.

Description

Technical Field

[0001] This invention relates to the field of digital interaction technology, specifically to a digital NPC character emotion expression and interaction system. Background Technology

[0002] With the rapid development and popularization of augmented reality and digital human technologies, digital NPC characters have gradually become important objects of user interaction in AR applications. As a core element affecting user immersion and experience quality, the emotional expression and interaction capabilities of NPC characters directly determine user satisfaction in the augmented reality environment. Therefore, it is essential to build a digital NPC character emotional expression and interaction system with emotional perception and dynamic response capabilities.

[0003] In existing technologies, most digital NPC character interaction systems rely on preset fixed dialogue scripts and standardized facial expression libraries, which cannot be dynamically adjusted according to the user's real-time emotional state. This results in the NPC character's emotional expression exhibiting mechanical and patterned characteristics. Although some existing technologies have introduced voice emotion recognition or facial expression analysis, due to their single data collection dimension, it is difficult to accurately judge the user's complex emotional state. In the ever-changing interaction scenarios, using only single-modal data for emotion recognition has a large error. When faced with subtle changes in the user's emotions, if only simple emotion classification is used, it is easy to produce inappropriate results.

[0004] Existing technologies lack the ability to jointly collect, fuse, analyze, and coordinate multimodal data such as user facial micro-expressions, body posture, and voice prosody. They also lack a mechanism to recognize the natural pauses and rhythms between NPC character voice output and body movements, making it impossible to achieve coordinated coordination between voice prosody and body movements. Furthermore, existing systems often generate complete response content at once and then play it continuously, resulting in defects such as the inability to adjust midway based on real-time user feedback, stiff interaction rhythm, and insufficient naturalness. In terms of perceiving the trend of user emotional changes, existing technologies mainly rely on overall evaluation after the interaction ends, failing to capture changes in the user's emotional state in real time during the interaction and dynamically adjust the NPC character's expression accordingly. This leads to a significant gap between the interaction effect and the user's actual emotional needs. At the same time, they lack adaptive response strategies for different combinations of emotional characteristics and cannot dynamically select the optimal response emotion type based on the user's multiple dominant emotional characteristics. Summary of the Invention

[0005] The purpose of this invention is to provide a digital NPC character emotion expression and interaction system, which solves the problems existing in the background technology.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A digital NPC character emotion expression and interaction system includes: a real-time user state perception module, used to collect facial micro-expression data, body posture data and audio data of the user in the current language segment when using AR devices to communicate with NPC characters, and to analyze the emotional state characteristics of the user in the current language segment based on these data.

[0007] The complete interaction scheme generation module is used to obtain the preset personality model parameter set of NPC characters and the user's historical memory data from the local database, and generate a complete feedback interaction scheme for the NPC characters in the current language segment based on the user's emotional state characteristics in the current language segment.

[0008] The interval point identification and segmentation module is used to perform semantic pause analysis and action connection analysis on the complete feedback interaction scheme of NPC characters, identify each natural interval point and its corresponding interval duration in the complete interaction scheme of NPC characters, and segment the complete interaction scheme into the interaction segments of NPC characters accordingly.

[0009] The dynamic fine-tuning execution module is used to execute each interaction segment of the NPC character in sequence. If the execution is not interrupted by the user, it pauses for the corresponding interval at each natural interval point, analyzes the changing trend of the user's emotional state characteristics at each natural interval point, and adjusts the feedback interaction scheme of the NPC character for the next interaction segment at each natural interval point before continuing execution. After execution is completed, the emotional memory evolution module is executed. If the user interrupts, the real-time user state perception module is re-executed.

[0010] The Emotional Memory Evolution Module is used to record the interaction execution data between the user and the NPC character during this interaction. After the above data is structured, it is stored in the local database to form the user's historical memory data.

[0011] The beneficial effects of this invention are as follows: This invention constructs a multimodal, three-dimensional user emotional state feature analysis mechanism, improving the accuracy and dimensional coverage of user emotional state recognition. By combining target parameters of various emotional features and calculating initial score values, and adjusting the action amplitude values, the target score value is obtained. This comprehensively captures the user's explicit emotional expression and implicit emotional tendency, forming more accurate and delicate emotional state features, which facilitates subsequent NPC role response strategy matching. By retrieving and matching each allowed response emotional type and comprehensively evaluating it, the target response emotional type is obtained, realizing a response mechanism that comprehensively considers multiple emotional features. This invention can improve the adaptability of NPC role responses and the satisfaction of user interaction experience. At the same time, this invention can make the NPC role's voice output and body movements present a natural and smooth rhythm. Based on the real-time emotional state feature change trend analysis mechanism at each natural interval point, it can continuously monitor user emotional changes during the interaction process. By calculating the emotional change trend deviation, the speech rate, volume, and action amplitude parameters of the next interaction segment are dynamically adjusted, realizing real-time adaptive optimization of NPC role interaction expression. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of the system modules of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] Reference Figure 1 As shown, the present invention provides a digital NPC character emotion expression and interaction system, including: a user status real-time perception module, a complete interaction scheme generation module, an interval point identification and segmentation module, a dynamic fine-tuning execution module, and an emotion memory evolution module.

[0016] It should be noted that the real-time user status perception module is connected to the complete interaction scheme generation module, the complete interaction scheme generation module is connected to the interval point recognition and segmentation module, the interval point recognition and segmentation module is connected to the dynamic fine-tuning execution module, and the dynamic fine-tuning execution module is connected to the emotional memory evolution module.

[0017] The real-time user status perception module is used to collect facial micro-expression data, body posture data, and audio data of users when they interact with NPC characters using AR devices, and to analyze the emotional state characteristics of users in the current language segment based on this data.

[0018] In one specific embodiment, facial micro-expression data, body posture data, and audio data of the user in the current speech segment are collected. The specific method is as follows: by using an AR device worn by the user, the facial micro-expression data, body posture data, and audio data of the user in the current speech segment are obtained.

[0019] It should be noted that facial micro-expression data includes: facial image frame sequences; body posture data includes: three-dimensional coordinate sequences of each joint at each monitoring time point; and audio data includes: audio data such as volume, pitch, and speech-text at each monitoring time point.

[0020] In a specific embodiment of the present invention, the emotional state characteristics of the user in the current language segment are analyzed by means of: based on the user's facial micro-expression data in the current language segment, the activation intensity of the muscle motor units in each facial region of the user at each monitoring time point is detected by a facial motion coding network.

[0021] In one specific embodiment, the activation intensity of muscle motor units in each facial region at each monitoring time point is detected by the following method: the infrared camera of the AR device collects a sequence of facial image frames of the user at a preset sampling frequency, the facial image frames at each monitoring time point are input into the facial motion coding network, the facial motion coding network performs facial region segmentation on the facial image frames, identifies the user's eyebrow region, eye area, nose wing region, corner of mouth region and jaw region, and detects the muscle motor units in each facial region at each monitoring time point, quantifies the displacement amplitude of each muscle motor unit in each facial region at each monitoring time point, statistically obtains the comprehensive displacement amplitude of the muscle motor units in each facial region at each monitoring time point, and uses it as the activation intensity of the muscle motor units in each facial region at each monitoring time point.

[0022] Based on the user's body posture data in the current language segment, extract the three-dimensional coordinate sequence of the user at each joint point at each monitoring time point.

[0023] Based on the user's audio data in the current language segment, analyze the user's time-Mel frequency cepstral coefficients, mean fundamental frequency, rate of change of fundamental frequency, and mean short-time energy at each monitoring time point.

[0024] In one specific embodiment, the Mel frequency cepstral coefficients, mean fundamental frequency, rate of change of fundamental frequency, and mean short-time energy of the user at each monitoring time point are analyzed. The specific method is as follows: existing audio processing technology is relatively mature. By using existing audio processing technology and combining it with the user's audio data in the current language segment, the Mel frequency cepstral coefficients, mean fundamental frequency, rate of change of fundamental frequency, and mean short-time energy of the user at each monitoring time point can be obtained.

[0025] Based on the user's Mel frequency cepstral coefficients, mean fundamental frequency, rate of change of fundamental frequency, and mean short-time energy at each monitoring time point, and combined with the user's muscle motor unit activation intensity in each facial region and the three-dimensional coordinate sequence at each joint point at each monitoring time point, multimodal feature fusion processing is performed to output the user's emotional state features in the current language segment.

[0026] In a specific embodiment of the present invention, the emotional state characteristics of the user in the current language segment are output. The specific method is as follows: the target parameter combination of each emotional characteristic is obtained from the local database. Based on the user's Mel frequency cepstral coefficient, fundamental frequency mean, fundamental frequency change rate and short-time energy mean at each monitoring time point, and combined with the activation intensity of the muscle motor unit in each facial region at each monitoring time point, the parameters are combined to obtain the value of each parameter in the target parameter combination of each emotional characteristic at each monitoring time point. Based on this, the initial score value of each emotional characteristic of the user at each monitoring time point is calculated.

[0027] It should be noted that the local database is used to store the target parameter combination of each emotional feature, the parameter calculation formula of each emotional feature, the preset personality model parameter set of NPC characters and the user's historical memory data, the dialogue generation model, the preset basic score comparison value of each emotional feature, the score threshold of the dominant emotional feature, the allowed response emotion types corresponding to each emotional feature type, the speech rate adjustment ratio, tone adjustment ratio and volume adjustment ratio corresponding to each semantic unit type, the local adjustment parameters corresponding to each emotional emphasis word and tone word, the body movement generation model, the semantic encoding model, the basic interval duration corresponding to each type of ending punctuation, the adjustment interval duration value corresponding to each buffer adjustment value range, and the speech rate adjustment coefficient, volume adjustment coefficient and movement amplitude adjustment coefficient corresponding to each emotional change trend deviation range.

[0028] It should also be noted that the various emotional characteristics include: anger, sadness and pleasure, etc. The target parameter combination, such as the target of the anger emotional characteristic, includes the fundamental frequency mean, short-time energy mean, eyebrow region muscle motor unit activation intensity and nose wing region muscle motor unit activation intensity. The target parameter combination can be set by researchers according to actual research. This invention only provides one design idea.

[0029] In one specific embodiment, the initial score values ​​of each emotional characteristic of the user at each monitoring time point are calculated. The specific method is as follows: obtain the parameter calculation formula of each emotional characteristic from the local database, and substitute the values ​​of each parameter in the target parameter combination of each emotional characteristic of the user at each monitoring time point into the corresponding parameter calculation formula. For example, the parameter calculation formula for the anger emotional characteristic is: ,in These are the initial scores for the anger characteristic at each monitoring time point. This represents the short-time energy average at each monitoring time point. A preset comparison value is set for the short-term energy average. The mean of the fundamental frequency at each monitoring time point. A preset comparison value is set for the fundamental frequency mean. The activation intensity of the eyebrow region muscle motor units on the downward vector at each monitoring time point. This represents the activation intensity of the nasal alar region muscle motor units on the outward vector. It is a natural constant. Number each monitoring time point. , The parameters are positive integers greater than 2, and the calculation formulas for each emotional characteristic can be set by researchers based on actual research.

[0030] Based on the user's three-dimensional coordinate sequence at each key point at each monitoring time point, the user's action amplitude value at each monitoring time point is evaluated, and the initial score value of each emotional feature of the user at each monitoring time point is adjusted accordingly, so as to obtain the target score value of each emotional feature of the user at each monitoring time point.

[0031] In one specific embodiment, the method for evaluating the user's motion amplitude value at each monitoring time point is as follows: based on the three-dimensional coordinate sequence of the user at each joint point at each monitoring time point, calculate the displacement distance of each joint point between adjacent monitoring time points, and sum the displacement distances of each joint point by weight to obtain the overall displacement of the user at each monitoring time point, which is used as the user's motion amplitude value at each monitoring time point.

[0032] In one specific embodiment, the target score value of each emotional feature of the user at each monitoring time point is obtained by multiplying the user's action amplitude value at each monitoring time point with the user's initial score value of each emotional feature at each monitoring time point.

[0033] The target score of each emotional feature of the user at each monitoring time point is used as the emotional state feature of the user in the current language segment.

[0034] The complete interaction scheme generation module is used to obtain the preset personality model parameter set of NPC characters and the user's historical memory data from the local database, and generate a complete feedback interaction scheme for the NPC characters in the current language segment based on the user's emotional state characteristics in the current language segment.

[0035] It should be noted that the preset personality model parameter set includes: expression style parameter set, and historical memory data, including: historical dialogue context records in each historical round.

[0036] It should also be noted that the set of expression style parameters includes: initial speech rate, initial pitch, and initial volume values ​​for each response emotion type.

[0037] In a specific embodiment of the present invention, a complete feedback interaction scheme for NPC characters in the current language paragraph is generated. The specific method is as follows: based on the preset personality model parameter set of NPC characters, the expression style parameter set of NPC characters is extracted.

[0038] Based on the user's emotional state characteristics in the current language segment, the target response emotional type of the NPC character is obtained.

[0039] Based on the user's historical memory data, extract the historical dialogue context records of the user in each historical round.

[0040] The semantic text content of the user in the current language segment is obtained, and combined with the user's historical dialogue context records in each historical round and the target response emotion type of the NPC character, the feedback voice text of the NPC character is analyzed.

[0041] In one specific embodiment, the semantic text content of the user in the current language segment is obtained by extracting it using existing text parsing techniques based on the user's audio data in the current language segment.

[0042] In one specific embodiment, the feedback voice text of the NPC character is analyzed and obtained. The specific method is as follows: the dialogue generation model is obtained from the local database. The semantic text content of the user in the current language segment, the historical dialogue context record of the user in each historical round, and the target response emotion type of the NPC character are input into the dialogue generation model as input parameters. The dialogue generation model understands the user's expressive intention based on the semantic text content, maintains the dialogue coherence by combining the historical dialogue context record, and selects the corresponding language style and vocabulary expression according to the target response emotion type to generate the output feedback voice text of the NPC character. The dialogue generation model generates data through a large number of dialogue models.

[0043] Based on the NPC character's feedback voice text, target response emotion type, and expression style parameter set, generate the NPC character's feedback voice prosodic parameter sequence and feedback body movement control sequence.

[0044] The feedback voice text, feedback voice prosody parameter sequence, and feedback body movement control sequence of NPC characters are combined to form a complete feedback interaction scheme for NPC characters.

[0045] In a specific embodiment of the present invention, the target response emotion type of the NPC character is obtained by: calculating the emotion deviation based on the target score value of each emotion feature of the user at each monitoring time point, and obtaining the comprehensive score value of each emotion feature of the user in the current language segment.

[0046] In one specific embodiment, the comprehensive score value of each emotional feature of the user in the current language segment is obtained by: obtaining the preset basic score comparison value of each emotional feature from the local database. ,in These are represented by numbers for each emotional characteristic. , A positive integer greater than 2, based on the target score value of each emotional characteristic of the user at each monitoring time point. Calculate the comprehensive score of the user's emotional features in the current language segment. .

[0047] Based on the user's comprehensive score for each emotional feature in the current language segment, the type of each dominant emotional feature of the user is determined, and the proportion of each dominant emotional feature of the user is evaluated.

[0048] In one specific embodiment, the type of each dominant emotional feature of the user is determined, and the proportion of each dominant emotional feature of the user is evaluated. The specific method is as follows: obtain the dominant emotional feature scoring threshold from the local database, compare the comprehensive score value of each emotional feature of the user in the current language segment with the dominant emotional feature scoring threshold, if the comprehensive score value of a certain emotional feature of the user in the current language segment is greater than the dominant emotional feature scoring threshold, then mark the emotional feature as the user's dominant emotional feature, thereby determining the user's dominant emotional features and their types, extracting the comprehensive score value of each dominant emotional feature of the user in the current language segment, and calculating the proportion of each dominant emotional feature of the user.

[0049] Retrieve each allowed emotional response type from the local database, and search for each allowed emotional response type that matches the user's dominant emotional characteristics.

[0050] It should be noted that the researchers pre-configured a correspondence table between various emotional trait types and allowed response emotion types. For each emotional trait type, multiple response emotion types that NPC characters can adopt are set. For example, the allowed response emotion types corresponding to the anger trait type include soothing, calming, and encouraging, while the allowed response emotion types corresponding to the sadness trait type include caring, encouraging, and companionship, and the allowed response emotion types corresponding to the joy trait type include companionship, enthusiasm, and calmness.

[0051] Based on the user's dominant emotional characteristics and the allowable response emotional types, and combined with the proportion of each dominant emotional characteristic, the target response emotional type of the NPC character is evaluated.

[0052] In one specific embodiment, the method for evaluating the target response emotion type of an NPC character is as follows: based on the user's allowed response emotion types for each dominant emotion feature, the total number of occurrences of each allowed response emotion type for the user is counted, and the total number of occurrences of each allowed response emotion type for each dominant emotion feature is mapped to the total number of occurrences of each allowed response emotion type for each dominant emotion feature across all dominant emotion features. This value is multiplied by the proportion of each dominant emotion feature for the user to obtain the parameter-adjusted value of the number of occurrences of each allowed response emotion type for each dominant emotion feature across all dominant emotion features. The parameter-adjusted values ​​of the number of occurrences of each allowed response emotion type for each dominant emotion feature are then summed to obtain the selection score value of each allowed response emotion type for the user. The maximum allowed response emotion type in the selection score system is selected as the target response emotion type for the NPC character.

[0053] In a specific embodiment of the present invention, the method for generating the feedback voice prosody parameter sequence and feedback body movement control sequence of the NPC character is as follows: semantic segmentation processing is performed based on the feedback voice text of the NPC character to obtain each semantic unit of the NPC character and its text content.

[0054] In one specific embodiment, the semantic units and their text content of the NPC character are obtained by the following method: the existing semantic segmentation processing technology is relatively mature. By using the existing semantic segmentation processing technology and combining it with the feedback voice text of the NPC character, the semantic units and their text content of the NPC character can be obtained.

[0055] Based on the expression style parameter set of NPC characters, the initial speech rate, initial tone and initial volume values ​​of NPC characters in each response emotion type are extracted, and combined with the target response emotion type of NPC characters, the initial speech rate, initial tone and initial volume values ​​of NPC characters are determined.

[0056] Based on the initial speech rate, initial pitch, and initial volume of the NPC character, and combined with the text content of each semantic unit of the NPC character, speech adjustment is performed to obtain the set speech rate, set pitch, and set volume of the NPC character in each semantic unit. These values ​​are then combined in the order of the semantic units of the NPC character to form the feedback speech prosodic parameter sequence of the NPC character.

[0057] In one specific embodiment, the method for setting the speech rate, tone, and volume of an NPC character in each semantic unit is as follows: Based on the text content of each semantic unit of the NPC character, semantic type analysis is performed to determine whether the semantic type of each semantic unit is declarative, interrogative, exclamatory, or imperative, etc. The speech rate adjustment ratio, tone adjustment ratio, and volume adjustment ratio corresponding to the type of each semantic unit are obtained from the local database. The initial speech rate, tone, and volume values ​​of the NPC character are multiplied by the corresponding adjustment ratios to obtain the adjusted speech rate, tone, and volume values ​​of the NPC character in each semantic unit. Simultaneously, based on the text content of each semantic unit of the NPC character, emotional emphasis words and tone words are identified. The local adjustment parameters corresponding to each emotional emphasis word and tone word are obtained from the local database. The speech rate, tone, and volume values ​​at the locations of the emotional emphasis words and tone words in each semantic unit are locally adjusted to obtain the set speech rate, tone, and volume values ​​of the NPC character in each semantic unit.

[0058] The model for generating body movements is obtained from a local database. Based on the semantic content of each semantic unit of the NPC character and the target response emotion type of the NPC character, the model is input into the body movement generation model to generate body movement fragments and movement amplitudes corresponding to each semantic unit of the NPC character.

[0059] It should be noted that the limb motion generation model is a pre-trained neural network model. Its input parameters include the semantic content of the semantic unit and the emotional type of the target response. Its output parameters are the corresponding limb motion fragments and motion amplitude. The limb motion generation model contains a semantic encoding layer, an emotion encoding layer, and a motion decoding layer. The semantic encoding layer is used to encode the semantic content of the semantic unit into a semantic feature vector. The emotion encoding layer is used to encode the emotional type of the target response into an emotion feature vector. The motion decoding layer is used to decode and generate the corresponding limb motion fragments and motion amplitude based on the semantic feature vectors and emotion feature vectors. The limb motion fragments include the position coordinates and rotation angles of each skeletal control point in each time frame, and the motion amplitude represents the overall motion amplitude coefficient of the limb motion.

[0060] The body movement fragments and movement amplitudes corresponding to each semantic unit of the NPC character are spliced ​​together in chronological order to form the feedback body movement control sequence of the NPC character.

[0061] The interval point identification and segmentation module is used to perform semantic pause analysis and action connection analysis on the complete feedback interaction scheme of NPC characters, identify each natural interval point and its corresponding interval duration in the complete interaction scheme of NPC characters, and segment the complete interaction scheme into the interaction segments of NPC characters accordingly.

[0062] In a specific embodiment of the present invention, the method for identifying each natural interval point and its corresponding interval duration in the complete interaction scheme of the NPC character is as follows: based on the order of each semantic unit of the NPC character, the boundary position between adjacent semantic units of the NPC character is determined as each natural interval point, thereby obtaining each natural interval point in the complete interaction scheme of the NPC character.

[0063] Based on the text content of the semantic units adjacent to each natural interval point of the NPC character, analyze the semantic correlation between the adjacent semantic units of the NPC character at each natural interval point.

[0064] In one specific embodiment, the semantic correlation between adjacent semantic units of an NPC character at each natural interval is analyzed. The specific method is as follows: extract the text content of the semantic unit preceding and following each natural interval of the NPC character; obtain the semantic coding model from the local database; input the text content of the semantic unit preceding and following each natural interval of the NPC character into the semantic coding model to obtain the semantic feature vector of the semantic unit preceding and following each natural interval of the NPC character; calculate the cosine similarity between the semantic feature vector of the semantic unit preceding and following each natural interval of the NPC character; and use the cosine similarity as the semantic correlation between adjacent semantic units of the NPC character at each natural interval.

[0065] Based on the limb action segments corresponding to each semantic unit of the NPC character, the ending posture and starting posture of the adjacent limb action segments before and after each natural interval point of the NPC character are extracted, and the posture transition amplitude between the adjacent limb action segments before and after each natural interval point of the NPC character is calculated.

[0066] In one specific embodiment, the posture transition amplitude between adjacent limb action segments of an NPC character before and after each natural interval point is calculated. The specific method is as follows: extract the position coordinates and rotation angles of each skeletal control point in the last frame of the limb action segment before each natural interval point as the ending posture; extract the position coordinates and rotation angles of each skeletal control point in the first frame of the limb action segment after each natural interval point as the starting posture; calculate the difference in position coordinates and rotation angles of each skeletal control point between the ending posture and the starting posture of the adjacent limb action segments of the NPC character at each natural interval point; and perform a weighted summation of the difference in position coordinates and rotation angles of each skeletal control point to obtain the posture transition amplitude between adjacent limb action segments of the NPC character before and after each natural interval point.

[0067] Based on the text content of the semantic unit preceding each natural interval of an NPC character, extract the type of the ending punctuation mark of the semantic unit preceding each natural interval of an NPC character.

[0068] The basic interval duration corresponding to each type of ending punctuation is obtained from the local database. Based on the type of ending punctuation of the semantic unit preceding each natural interval of the NPC character, the semantic correlation between adjacent semantic units, and the posture transition amplitude between adjacent limb action segments, the corresponding interval duration of each natural interval of the NPC character is calculated.

[0069] It should be noted that the basic interval duration corresponding to each type of punctuation mark at the end is set by researchers based on actual experimental conditions, and there is no fixed value.

[0070] In one specific embodiment, the corresponding interval duration of each natural interval point of an NPC character is calculated as follows: based on the type of the ending punctuation mark of the semantic unit preceding each natural interval point of the NPC character, the basic interval duration of each natural interval point of the NPC character is mapped; and based on the semantic correlation between adjacent semantic units of each natural interval point of the NPC character... The amplitude of posture transition between adjacent limb movement segments Multiply, where This is represented by the number of each natural interval point. , A positive integer greater than 2, used to obtain the buffer adjustment value between adjacent semantic units at each natural interval point of the NPC character. The adjustment interval duration value corresponding to each buffer adjustment value range is obtained from the local database, mapped to the adjustment interval duration value of each natural interval point of the NPC character, and added to the base interval duration of each natural interval point of the NPC character to obtain the corresponding interval duration of each natural interval point of the NPC character.

[0071] In one specific embodiment, the complete interaction scheme is divided into various interaction segments of the NPC character. The specific method is as follows: based on the natural interval points in the complete interaction scheme of the NPC character, the complete interaction scheme is divided according to the location of the natural interval points, thereby obtaining various interaction segments of the NPC character.

[0072] The dynamic fine-tuning execution module is used to execute each interaction segment of the NPC character in sequence. If the execution is not interrupted by the user, it pauses for the corresponding interval at each natural interval point, analyzes the changing trend of the user's emotional state characteristics at each natural interval point, and adjusts the feedback interaction scheme of the NPC character for the next interaction segment at each natural interval point before continuing execution. After execution is completed, the emotional memory evolution module is executed. If the user interrupts, the real-time user state perception module is re-executed.

[0073] In a specific embodiment of the present invention, the user's emotional state characteristics change trend is analyzed, and the feedback interaction scheme of the NPC character in the next interaction segment at each natural interval point is adjusted accordingly before continuing execution. The specific method is as follows: during the interval time of the NPC character at each natural interval point, the real-time emotional state characteristics of the character during the interval time of each natural interval point are analyzed in the same way as the method of analyzing the user's emotional state characteristics in the current language segment.

[0074] The user's emotional state features in the current language segment are used as the baseline emotional state features. The real-time emotional state features of the character at each natural interval are compared with the baseline emotional state features to perform emotional feature deviation analysis. The score offset values ​​of each emotional feature of the user at each natural interval are calculated and used as the trend of the user's emotional state feature changes.

[0075] In one specific embodiment, the rating offset values ​​of each emotional feature of the user at each natural interval point are calculated. The specific method is as follows: based on the comprehensive rating value of each emotional feature of the user in the current language segment, and based on the real-time emotional state features of the character at each natural interval point, the real-time rating values ​​of each emotional feature of the character at each monitoring time point at each natural interval point are extracted from the real-time emotional state features. Similarly, the real-time comprehensive rating value of the target rating value of each emotional feature of the character at each natural interval point is calculated. The difference between the real-time comprehensive rating value of each emotional feature in the real-time emotional state features and the comprehensive rating value of the corresponding emotional feature in the benchmark emotional state features is calculated to obtain the rating offset value of each emotional feature of the user at each natural interval point.

[0076] Based on the changing trends of users' emotional state characteristics, the speech rate adjustment coefficient, audio adjustment coefficient, and action amplitude adjustment coefficient of users in the next interaction segment at each natural interval are analyzed. Based on this, the feedback interaction scheme of NPC characters in the next interaction segment at each natural interval is adjusted, and the next interaction segment is executed after the adjustment is completed.

[0077] In a specific embodiment of the present invention, the speech rate adjustment coefficient, audio adjustment coefficient, and motion amplitude adjustment coefficient of the user in the next interactive segment at each natural interval are analyzed. The specific method is as follows: based on the positive or negative direction of the rating offset value of each emotional feature of the user at each natural interval, it is determined whether the change trend of each emotional feature of the user at each natural interval is positive or negative. Then, the proportion of positively enhanced emotional features and the proportion of negatively enhanced emotional features of the user at each natural interval are counted, and the deviation of the user's emotional change trend at each natural interval is determined accordingly.

[0078] In one specific embodiment, the deviation of the user's emotional change trend at each natural interval is determined by the following method: based on the proportion of positively enhanced emotional features and the proportion of negatively enhanced emotional features at each natural interval, the proportion of positively enhanced emotional features is subtracted from the proportion of negatively enhanced emotional features to obtain the deviation of the user's emotional change trend at each natural interval. The value of the deviation ranges from negative one to positive one. When the deviation is positive, it indicates that the user's overall emotion tends to be positively enhanced; when it is negative, it indicates that the user's overall emotion tends to be negatively enhanced; and when it is close to zero, it indicates that the user's emotion is in a stable state.

[0079] The speech rate adjustment coefficient, volume adjustment coefficient, and motion amplitude adjustment coefficient corresponding to each range of emotional change trend deviation are obtained from the local database. Based on the user's emotional change trend deviation at each natural interval, the speech rate adjustment coefficient, volume adjustment coefficient, and motion amplitude adjustment coefficient for the next interaction segment at each natural interval are mapped to the user.

[0080] It should be noted that the greater the deviation in the emotional change trend, the greater the corresponding speech rate adjustment coefficient, volume adjustment coefficient, and movement amplitude adjustment coefficient. These values ​​can be set by researchers based on the actual experimental model.

[0081] In a specific embodiment of the present invention, the feedback interaction scheme of the NPC character in the next interactive segment at each natural interval is adjusted. The specific method is as follows: based on the feedback speech prosody parameter sequence and feedback body movement control sequence of the NPC character, the original speech rate value, original volume value, and original movement amplitude of the NPC character in the next interactive segment at each natural interval are extracted, and multiplied by the speech rate adjustment coefficient, volume adjustment coefficient, and movement amplitude adjustment coefficient of the user in the next interactive segment at each natural interval, respectively, to obtain the appropriate speech rate value, appropriate volume value, and appropriate movement amplitude of the user in the next interactive segment at each natural interval. These values ​​are then input into the feedback interaction scheme of the NPC character in the next interactive segment at each natural interval for adjustment.

[0082] The Emotional Memory Evolution Module is used to record the interaction execution data between the user and the NPC character during this interaction. After the above data is structured, it is stored in the local database to form the user's historical memory data.

[0083] This invention constructs a multimodal, three-dimensional user emotional state feature analysis mechanism to improve the accuracy and dimensional coverage of user emotional state recognition. By combining target parameters of various emotional features and calculating initial scores, and adjusting the action amplitude values, a target score value is obtained. This mechanism can comprehensively capture users' explicit emotional expressions and implicit emotional tendencies, forming more accurate and delicate emotional state features. This facilitates subsequent NPC role response strategy matching. By retrieving and matching each allowed response emotional type and comprehensively evaluating them, the target response emotional type is obtained, realizing a response mechanism that comprehensively considers multiple emotional features. This invention can improve the adaptability of NPC role responses and the satisfaction of user interaction experience. At the same time, this invention can make the NPC role's voice output and body movements present a natural and smooth rhythm. Based on the real-time emotional state feature change trend analysis mechanism at each natural interval point, it can continuously monitor user emotional changes during the interaction process. By calculating the deviation of emotional change trends, it dynamically adjusts the speech rate, volume, and action amplitude parameters of the next interaction segment, realizing real-time adaptive optimization of NPC role interaction expression.

[0084] All formulas in this manual are dimensionless and calculated numerically. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0085] The above content is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined by the present invention, and all such modifications and additions should fall within the protection scope of the present invention.

Claims

1. A digital NPC character emotion expression and interaction system, characterized in that, include: The real-time user status perception module is used to collect facial micro-expression data, body posture data and audio data of users in the current language segment when interacting with NPC characters using AR devices, and to analyze the emotional state characteristics of users in the current language segment based on these data. The complete interaction scheme generation module is used to obtain the preset personality model parameter set of NPC characters and the user's historical memory data from the local database, and generate a complete feedback interaction scheme for the NPC characters in the current language segment based on the user's emotional state characteristics in the current language segment. The interval point identification and segmentation module is used to perform semantic pause analysis and action connection analysis on the complete feedback interaction scheme of NPC characters, identify each natural interval point and its corresponding interval duration in the complete interaction scheme of NPC characters, and segment the complete interaction scheme into the interaction segments of NPC characters accordingly. The dynamic fine-tuning execution module is used to execute each interaction segment of the NPC character in sequence. If the execution is not interrupted by the user, it pauses for the corresponding interval at each natural interval point, analyzes the changing trend of the user's emotional state characteristics at each natural interval point, and adjusts the feedback interaction scheme of the NPC character for the next interaction segment at each natural interval point before continuing execution. After the execution is completed, the emotional memory evolution module is executed. If the user interrupts, the real-time user state perception module is re-executed. The Emotional Memory Evolution Module is used to record the interaction execution data between the user and the NPC character during this interaction. After the above data is structured, it is stored in the local database to form the user's historical memory data.

2. The digital NPC character emotion expression and interaction system according to claim 1, characterized in that, The specific method for analyzing the user's emotional state characteristics in the current language segment is as follows: Based on the user's facial micro-expression data in the current speech segment, the activation intensity of muscle motor units in each facial region is detected by the facial motion coding network at each monitoring time point; Based on the user's body posture data in the current language segment, extract the three-dimensional coordinate sequence of the user at each joint point at each monitoring time point; Based on the user's audio data in the current language segment, analyze the user's time-frequency cepstral coefficients, mean fundamental frequency, rate of change of fundamental frequency, and mean short-time energy at each monitoring time point; Based on the user's Mel frequency cepstral coefficients, mean fundamental frequency, rate of change of fundamental frequency, and mean short-time energy at each monitoring time point, and combined with the user's muscle motor unit activation intensity in each facial region and the three-dimensional coordinate sequence at each joint point at each monitoring time point, multimodal feature fusion processing is performed to output the user's emotional state features in the current language segment.

3. The digital NPC character emotion expression and interaction system according to claim 2, characterized in that, The output obtains the user's emotional state characteristics in the current language segment, and the specific method is as follows: The target parameter combination of each emotional feature is obtained from the local database. Based on the user's Mel frequency cepstral coefficient, fundamental frequency mean, fundamental frequency change rate and short-time energy mean at each monitoring time point, and combined with the activation intensity of the muscle motor unit in each facial region at each monitoring time point, the parameters are combined to obtain the value of each parameter in the target parameter combination of each emotional feature at each monitoring time point. Based on this, the initial score value of each emotional feature of the user at each monitoring time point is calculated. Based on the user's three-dimensional coordinate sequence at each key point at each monitoring time point, the user's action amplitude value at each monitoring time point is evaluated, and the initial score value of each emotional feature of the user at each monitoring time point is adjusted accordingly, so as to obtain the target score value of each emotional feature of the user at each monitoring time point. The target score of each emotional feature of the user at each monitoring time point is used as the emotional state feature of the user in the current language segment.

4. The digital NPC character emotion expression and interaction system according to claim 3, characterized in that, The specific method for generating a complete feedback interaction scheme for NPC characters in the current language paragraph is as follows: Based on the preset personality model parameter set of NPC characters, extract the expression style parameter set of NPC characters; Based on the user's emotional state characteristics in the current language segment, the target response emotional type of the NPC character is obtained; Based on the user's historical memory data, extract the historical dialogue context records of the user in each historical round; Obtain the semantic text content of the user in the current language segment, and combine it with the user's historical dialogue context records in each historical round and the target response emotion type of the NPC character to analyze and obtain the feedback voice text of the NPC character; Based on the NPC character's feedback voice text, target response emotion type, and expression style parameter set, generate the NPC character's feedback voice prosodic parameter sequence and feedback body movement control sequence; The feedback voice text, feedback voice prosody parameter sequence, and feedback body movement control sequence of NPC characters are combined to form a complete feedback interaction scheme for NPC characters.

5. A digital NPC character emotion expression and interaction system according to claim 4, characterized in that, The specific method for obtaining the target response emotion type of the NPC character is as follows: Based on the target scores of each emotional feature of the user at each monitoring time point, the emotional bias is calculated to obtain the comprehensive score of each emotional feature of the user in the current language segment. Based on the comprehensive score of each emotional feature of the user in the current language segment, determine the type of each dominant emotional feature of the user, and evaluate the proportion of each dominant emotional feature of the user. Retrieve each allowed emotional response type from the local database, and search for each allowed emotional response type that matches the user's dominant emotional characteristics. Based on the user's dominant emotional characteristics and the allowable response emotional types, and combined with the proportion of each dominant emotional characteristic, the target response emotional type of the NPC character is evaluated.

6. The digital NPC character emotion expression and interaction system according to claim 4, characterized in that, The specific method for generating the feedback voice prosody parameter sequence and feedback body movement control sequence for NPC characters is as follows: Semantic segmentation is performed on the feedback voice text of NPC characters to obtain each semantic unit of the NPC character and its text content; Based on the expression style parameter set of NPC characters, the initial speech rate, initial tone and initial volume of NPC characters in each response emotion type are extracted, and combined with the target response emotion type of NPC characters, the initial speech rate, initial tone and initial volume of NPC characters are determined. Based on the initial speech rate, initial pitch, and initial volume of the NPC character, and combined with the text content of each semantic unit of the NPC character, speech adjustment is performed to obtain the set speech rate, set pitch, and set volume of the NPC character in each semantic unit. These values ​​are then combined in the order of the semantic units of the NPC character to form the feedback speech prosodic parameter sequence of the NPC character. The model for generating body movements is obtained from the local database. Based on the semantic content of each semantic unit of the NPC character and the target response emotion type of the NPC character, the model is input into the body movement generation model to generate body movement fragments and movement amplitudes of the NPC character in each semantic unit. The body movement fragments and movement amplitudes corresponding to each semantic unit of the NPC character are spliced ​​together in chronological order to form the feedback body movement control sequence of the NPC character.

7. A digital NPC character emotion expression and interaction system according to claim 6, characterized in that, The specific method for identifying each natural interval point and its corresponding interval duration in the complete interaction scheme for identifying NPC characters is as follows: Based on the order of the semantic units of the NPC character, the boundary positions between adjacent semantic units of the NPC character are determined as natural interval points, thereby obtaining the natural interval points in the complete interaction scheme of the NPC character. Based on the text content of the semantic units adjacent to each natural interval point of the NPC character, analyze the semantic correlation between the adjacent semantic units of the NPC character at each natural interval point; Based on the limb action segments corresponding to each semantic unit of the NPC character, extract the ending posture and starting posture of the adjacent limb action segments of the NPC character before and after each natural interval point, and calculate the posture transition amplitude between the adjacent limb action segments of the NPC character before and after each natural interval point. Based on the text content of the semantic unit preceding each natural interval of the NPC character, extract the ending punctuation type of the semantic unit preceding each natural interval of the NPC character. The basic interval duration corresponding to each type of punctuation mark is obtained from the local database. Based on the type of punctuation mark of the semantic unit preceding each natural interval point of the NPC character, the semantic correlation between adjacent semantic units, and the posture transition amplitude between adjacent limb action segments, the corresponding interval duration of each natural interval point of the NPC character is calculated.

8. A digital NPC character emotion expression and interaction system according to claim 2, characterized in that, The method for analyzing the changing trends of the user's emotional state characteristics and adjusting the feedback interaction scheme of the NPC character in the next interaction segment at each natural interval point before continuing execution is as follows: During the intervals between natural intervals of NPC characters, based on the method of analyzing the emotional state characteristics of users in the current language segment, the real-time emotional state characteristics of characters during the intervals between natural intervals are analyzed. The user's emotional state features in the current language segment are used as the baseline emotional state features. The real-time emotional state features of the character at each natural interval are compared with the baseline emotional state features to perform emotional feature deviation analysis. The score offset values ​​of each emotional feature of the user at each natural interval are calculated and used as the trend of the user's emotional state features. Based on the changing trends of users' emotional state characteristics, the speech rate adjustment coefficient, audio adjustment coefficient, and action amplitude adjustment coefficient of users in the next interaction segment at each natural interval are analyzed. Based on this, the feedback interaction scheme of NPC characters in the next interaction segment at each natural interval is adjusted, and the next interaction segment is executed after the adjustment is completed.

9. A digital NPC character emotion expression and interaction system according to claim 8, characterized in that, The specific method for analyzing the user's speech rate adjustment coefficient, audio adjustment coefficient, and motion amplitude adjustment coefficient in the next interactive segment at each natural interval is as follows: Based on the positive or negative direction of the rating offset values ​​of each emotional feature at each natural interval, it is determined whether the trend of the user's emotional feature at each natural interval is positive or negative. Then, the proportion of positively enhanced emotional features and the proportion of negatively enhanced emotional features at each natural interval are counted, and the deviation of the user's emotional trend at each natural interval is determined accordingly. The speech rate adjustment coefficient, volume adjustment coefficient, and motion amplitude adjustment coefficient corresponding to each range of emotional change trend deviation are obtained from the local database. Based on the user's emotional change trend deviation at each natural interval, the speech rate adjustment coefficient, volume adjustment coefficient, and motion amplitude adjustment coefficient for the next interaction segment at each natural interval are mapped to the user.

10. A digital NPC character emotion expression and interaction system according to claim 9, characterized in that, The specific method for adjusting the feedback interaction scheme of the NPC character at each natural interval point for the next interaction segment is as follows: Based on the NPC character's feedback speech prosody parameter sequence and feedback body movement control sequence, the original speech rate, original volume, and original movement amplitude of the NPC character in the next interaction segment at each natural interval are extracted. These values ​​are then multiplied by the user's speech rate adjustment coefficient, volume adjustment coefficient, and movement amplitude adjustment coefficient in the next interaction segment at each natural interval to obtain the appropriate speech rate, appropriate volume, and appropriate movement amplitude for the user in the next interaction segment at each natural interval. These values ​​are then input into the NPC character's feedback interaction scheme in the next interaction segment at each natural interval for adjustment.

Images