Virtual character interaction device with emotion feedback function
By using adaptive segmentation and hierarchical dimensionality reduction in bioelectric signal processing, combined with historical trajectory prediction and dynamic dictionary of emotional prototype cluster centers, natural and coherent virtual character interaction behaviors are generated. This solves the problem of rigid emotional feedback mechanisms in existing technologies and achieves delicate perception and accurate prediction of user emotions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-23
- Publication Date
- 2026-04-07
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing virtual character emotional feedback mechanisms cannot effectively model the continuous and ambiguous characteristics of emotions, and are difficult to adapt to individual differences in the emotional expression patterns of different users, resulting in a stiff and shallow interaction process.
The algorithm employs a signal segmentation module, a hierarchical dimensionality reduction module, an emotion prototype mapping module, an emotion evolution prediction module, and an action synthesis module. It adaptively segments bioelectrical signals, performs hierarchical dimensionality reduction and compression, and extracts emotion features. It uses the historical evolution trajectory of the center of the emotion prototype cluster to predict emotion changes and combines a dynamic emotion dictionary and a virtual character action library to generate a continuous flow of action instructions.
It achieves a delicate perception and accurate prediction of emotional states, and the virtual character can generate natural, coherent and logically connected interactive behaviors to adapt to the emotional expression patterns of different users.
Smart Images
Figure CN121807153A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of human-computer interaction and affective computing technology, specifically to a virtual character interaction device with emotional feedback function. Background Technology
[0002] In the field of human-computer emotional interaction, enabling virtual characters to accurately perceive and respond to user emotions in a human-like manner is key to enhancing immersion. Existing technologies primarily rely on extracting features from facial images, speech, or physiological signals and mapping them to a limited set of discrete emotion category labels using classification models. This approach treats emotional states as independent, static categories, failing to effectively model the continuous distribution and mixing characteristics of emotions in the feature space. Furthermore, fixed classification models struggle to adapt to individual differences in users' emotional expression patterns, resulting in limited ability to recognize complex, subtle, or transitional emotions, directly impacting the accuracy of subsequent feedback.
[0003] Existing emotional feedback mechanisms for virtual characters typically trigger pre-set, isolated actions or expressions directly based on the identified instantaneous emotional state. This mechanism is inherently reactive, focusing only on the "current moment" state while completely ignoring the temporal evolution of emotion as a dynamic process. Due to the lack of analysis of the historical trajectory of emotional states and prediction of future trends, the behavioral feedback of virtual characters exhibits a mechanical, disjointed feel, failing to form coherent emotional expressions with internal logic and natural transitions, resulting in a stiff and shallow interaction process.
[0004] A technical solution is needed to overcome the above-mentioned shortcomings. This solution should be able to more precisely characterize the continuous and ambiguous characteristics of emotions, and have the ability to model and predict the dynamic evolution of emotions, thereby driving virtual characters to produce more natural, coherent, and predictive anthropomorphic interactive behaviors. Summary of the Invention
[0005] The purpose of this invention is to provide a virtual character interaction device with emotional feedback function to solve the problems mentioned in the background art.
[0006] To achieve the above objectives, the present invention provides a virtual character interaction device with emotional feedback function, the device comprising: The signal segmentation module acquires the raw sequence of bioelectric signals from the user's facial region and adaptively segments the raw sequence of bioelectric signals into multiple signal segments using time windows as units. The hierarchical dimensionality reduction module performs hierarchical dimensionality reduction and compression on each signal segment to obtain the sentiment feature vector corresponding to each time window. The sentiment feature vector contains a set of geometric morphology descriptors for the signal segment. The emotion prototype mapping module inputs the emotion feature vector into a pre-trained emotion prototype mapping network, which matches the nearest emotion prototype cluster center based on the feature space distance. The emotion evolution prediction module predicts the possible evolution path of the center of the emotion prototype cluster in the next time window based on the historical evolution trajectory of the center of the emotion prototype cluster in the emotion space. Based on the possible evolution path, it retrieves the corresponding emotion semantic tags and intensity parameters from the dynamic emotion dictionary, which is organized in a tree structure. The motion synthesis module selects basic motion units from the virtual character motion library based on the emotional semantic tags and intensity parameters; it then splices and smoothly transitions the selected basic motion units in the time domain to generate a continuous motion instruction stream.
[0007] Preferably, the step of performing hierarchical dimensionality reduction and compression on each signal segment to obtain the sentiment feature vector corresponding to each time window includes: Perform empirical wavelet transform on each signal segment to decompose the multiple intrinsic modes contained in the signal segment; Calculate the distribution of envelope extrema for each intrinsic mode, extract the skewness, kurtosis and fractal dimension of the envelope extrema distribution, and construct a set of geometric morphology descriptors for the signal segment. The set of geometric morphology descriptors is compressed and encoded using a sparse autoencoder to remove redundant information and retain the main feature patterns, outputting a low-dimensional sentiment feature vector.
[0008] Preferably, the emotional prototype mapping network matches the nearest emotional prototype cluster center based on feature space distance, including: The emotional prototype mapping network maintains a codebook composed of multiple emotional prototype cluster centers formed by clustering historical emotional feature vectors; Calculate the cosine similarity between the input sentiment feature vector and the center of each sentiment prototype cluster in the codebook; Several emotion prototype cluster centers with cosine similarity exceeding a set threshold are selected as candidates, and the weighted distance between the input emotion feature vector and each candidate emotion prototype cluster center under Gaussian kernel function mapping is calculated. The candidate emotion prototype cluster center with the smallest weighted distance is determined as the nearest emotion prototype cluster center.
[0009] Preferably, predicting the possible evolution path of the emotional prototype cluster center in the next time window based on the historical evolution trajectory of the emotional prototype cluster center in the emotional space includes: Retrieve all time points in a historical window of a preset time length from the nearest center of the emotion prototype cluster; Connect the centers of historical emotional prototype clusters corresponding to the time points to form the historical evolution trajectory of the centers of these emotional prototype clusters in the emotional space; The historical evolution trajectory is extrapolated using a time series prediction model with an extrapolation step size of one time window to obtain the extrapolated emotional prototype cluster center coordinates. The direction of the line connecting the extrapolated emotional prototype cluster center coordinates and the current emotional prototype cluster center coordinates is defined as the possible evolution path.
[0010] Preferably, the step of retrieving the corresponding sentiment semantic tags and intensity parameters from the dynamic sentiment dictionary based on the possible evolution path includes: The dynamic sentiment dictionary takes the sentiment dimension as the root node, and divides the sentiment dimension into multiple sentiment category sub-nodes. Each sentiment category sub-node stores specific sentiment semantic tag leaf nodes. Based on the direction vector of the possible evolution path in the emotion space, it is mapped to the corresponding emotion dimension in the dynamic emotion dictionary; The intensity parameter is determined based on the ratio of the magnitude of the directional vector to the average fluctuation amplitude of the center of the emotional prototype cluster along its historical evolution trajectory. In the sentiment category sub-nodes mapped to the sentiment dimension, sentiment semantic label leaf nodes with pre-defined intensity ranges are matched according to the intensity parameter.
[0011] Preferably, the step of selecting basic action units from the virtual character action library based on the emotional semantic tags and intensity parameters includes: The basic motion unit is defined by a keyframe sequence and a joint angle change curve; The virtual character action library uses emotional semantic tags as the main index, and each emotional semantic tag is associated with a set of basic action units sorted by intensity parameters. Based on the determined emotional semantic tags, the corresponding basic action unit group is located in the virtual character action library; Interpolation is performed in the basic motion unit group based on the intensity parameters, and the two basic motion units that are closest to the intensity parameters are selected as reference units. The keyframe sequences and joint angle change curves of the two reference units are linearly blended to generate the final basic motion unit used for driving.
[0012] Preferably, the step of splicing and smoothing the selected basic action units in the time domain to generate a continuous action command stream includes: Determine the duration of each basic motion unit, and adjust the playback rate of the basic motion unit according to the time window length and motion rhythm parameters; At the junction of two adjacent basic motion units, extract the end pose keyframe of the previous basic motion unit and the start pose keyframe of the next basic motion unit. A transition keyframe sequence is inserted at the junction. The transition keyframe sequence is generated by a pose interpolation algorithm so that the joint angle of the virtual character changes smoothly from the end pose keyframe to the start pose keyframe. The adjusted basic motion units and the inserted transition keyframe sequences are encoded in chronological order into a timestamped motion instruction stream.
[0013] Preferably, the device further includes a feedback calculation module for performing: The motion command stream drives the skeletal model of the virtual character to move, and synchronously updates the pixel values of the areas related to the emotional semantic tags in the facial texture map of the virtual character. Real-time capture of user feedback bioelectric signals to virtual character movements, and calculation of the deviation of the feedback bioelectric signals from the center of the current driving emotion prototype cluster in the feature space; The matching weights of the emotional prototype mapping network are corrected online based on the deviation, and the historical evolution trajectory of the emotional prototype cluster center in the emotional space is updated.
[0014] Preferably, calculating the deviation of the feedback bioelectric signal from the center of the current driving emotion prototype cluster in the feature space includes: The real-time captured user feedback bioelectric signals are subjected to the same adaptive segmentation and hierarchical dimensionality reduction and compression processing as the original sequence of bioelectric signals from the user's facial region to obtain the feedback emotion feature vector. Calculate the Mahalanobis distance between the feedback emotion feature vector and the center of the emotion prototype cluster currently driving the virtual character; The Mahalanobis distance is standardized so that its value range falls between zero and one, thus obtaining the standardized deviation.
[0015] Preferably, the step of online correction of the matching weights of the emotion prototype mapping network based on the deviation includes: Set a deviation threshold; when the standardized deviation exceeds the deviation threshold, trigger an online correction process. The feedback emotion feature vector is used as a new data point and merged with the current driving emotion prototype cluster center and its historical evolution trajectory data; The centroid position of the emotional prototype cluster center is recalculated using the merged data, and the coordinates of the corresponding emotional prototype cluster center in the emotional prototype mapping network codebook are updated. Based on the updated coordinates of the emotion prototype cluster centers, the matching weight matrix of the relevant emotion prototype cluster centers in the emotion prototype mapping network is recalculated.
[0016] Compared with the prior art, the beneficial effects of the present invention are: The Emotion Prototype Mapping Network does not employ direct classification. Instead, it matches emotions based on the calculated emotion feature vectors within a predefined emotion feature space, measuring the spatial distance to the centers of each emotion prototype cluster. Each prototype cluster center represents a core feature typical of a certain emotion state. This distance-based soft matching mechanism better represents complex emotions at the boundaries or in mixed states of different emotion categories, handling the ambiguity and continuity of emotions. By measuring the distance to multiple prototype clusters, the similarity between the current emotion state and various typical emotions can be quantified, rather than simply making a hard, either-or decision. This allows the system to more subtly perceive subtle changes in user emotions and adapt to the distribution of different users' unique emotion expression patterns in the feature space, improving the accuracy and robustness of emotion state discrimination.
[0017] By analyzing the movement trajectory of the center of the emotion prototype cluster within a continuous time window, the system can model the dynamic change pattern of a user's emotional state, and thus predict its possible development direction and path in the next moment. This elevates the system from merely identifying the "current state" to predicting the "state evolution." Combined with a dynamic emotion dictionary organized in a tree structure, built based on the semantic similarity and hierarchical relationships between emotion categories, the system can perform semantic retrieval and navigation within the emotion tree according to the predicted evolution path, accurately locating the corresponding emotional semantic tags and intensity parameters. This mechanism enables virtual characters to understand the trends and logic of emotional changes, thereby incorporating action elements that conform to the predicted trends into feedback in advance, achieving a smooth transition and natural connection in emotional expression, and generating coherent interactive behaviors with logical connections and emotional development context. Attached Figure Description
[0018] Figure 1 This is a schematic diagram illustrating the working principle of the virtual character interaction device with emotional feedback function described in this invention. Figure 2 This is a flowchart of hierarchical dimensionality reduction and feature extraction; Figure 3 A flowchart for predicting the path of emotional evolution; Figure 4 Comparison of blending effects for keyframe poses of virtual character motion units; Figure 5 This is a time-domain fluctuation diagram of signal strength. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] Please see Figure 1 This invention provides a virtual character interaction device with emotional feedback function. The device includes: acquiring the original sequence of bioelectrical signals from the user's facial region; adaptively segmenting the signal into multiple signal segments in units of time windows; performing hierarchical dimensionality reduction and compression on each signal segment to obtain an emotional feature vector, the emotional feature vector containing a set of geometric morphological descriptors of the signal segment; inputting the emotional feature vector into a pre-trained emotional prototype mapping network, which matches the nearest emotional prototype cluster center based on the feature space distance; predicting the possible evolution path of the emotional prototype cluster center in the next time window based on the historical evolution trajectory of the emotional prototype cluster center in the emotional space; retrieving the corresponding emotional semantic tags and intensity parameters from a dynamic emotional dictionary based on the possible evolution path; the dynamic emotional dictionary being organized in a tree structure; selecting basic action units from a virtual character action library based on the emotional semantic tags and intensity parameters; and splicing and smoothing the transition of the selected basic action units in the time domain to generate a continuous action command stream.
[0021] Example 1: See Figure 2 The signal segmentation module acquires the original sequence of bioelectrical signals from the user's facial region. It adaptively segments the original sequence of bioelectrical signals into multiple signal segments in units of time windows. The hierarchical dimensionality reduction module performs empirical wavelet transform on each signal segment to decompose multiple intrinsic modes contained in the signal segment. It calculates the distribution of the envelope extrema of each intrinsic mode and extracts the skewness, kurtosis and fractal dimension of the envelope extrema distribution to form a set of geometric morphology descriptors for the signal segment. The set of geometric morphology descriptors is compressed and encoded by a sparse autoencoder to remove redundant information and retain the main feature patterns, outputting a low-dimensional emotion feature vector. The Emotion Prototype Mapping Network maintains a codebook consisting of multiple emotion prototype cluster centers formed by clustering historical emotion feature vectors. It calculates the cosine similarity between the input emotion feature vector and each emotion prototype cluster center in the codebook, selects several emotion prototype cluster centers with a cosine similarity exceeding a set threshold as candidates, calculates the weighted distance between the input emotion feature vector and each candidate emotion prototype cluster center under Gaussian kernel function mapping, and determines the candidate emotion prototype cluster center with the smallest weighted distance as the nearest emotion prototype cluster center.
[0022] In practical implementation, the signal segmentation module acquires the raw sequence of bioelectrical signals from the user's facial region. This raw sequence is continuously recorded at a fixed sampling rate, with the time window length adaptively adjusted. The raw sequence is segmented into multiple signal segments, each containing a fixed number of sampling points. A hierarchical dimensionality reduction module performs an empirical wavelet transform on each signal segment, decomposing it into multiple intrinsic modes (EMS). The number of EMS depends on the frequency components of the signal segment. The distribution of the envelope extrema of each EMS is calculated, and the skewness, kurtosis, and fractal dimension of the distribution are extracted to form a set of geometric morphological descriptors for the signal segment. This set of geometric morphological descriptors is compressed and encoded using a sparse autoencoder. The sparse autoencoder has fewer neurons in its encoding layer than in the input layer. The compression removes redundant information from the set of geometric morphological descriptors while retaining the main feature patterns, outputting a low-dimensional emotional feature vector. The emotional prototype mapping network maintains a codebook composed of multiple emotional prototype cluster centers formed by clustering historical emotional feature vectors. The cosine similarity between the input emotional feature vector and each emotional prototype cluster center in the codebook is calculated. Several sentiment prototype cluster centers with cosine similarity exceeding a set threshold are selected as candidates. The weighted distance between the input sentiment feature vector and each candidate sentiment prototype cluster center under the Gaussian kernel function mapping is calculated. The formula for calculating the weighted distance is: in: This indicates that the input sentiment feature vector is related to the first... The weighted distance between the centers of each candidate emotional prototype cluster This represents the input sentiment feature vector. Indicates the first Center of a cluster of candidate emotional prototypes This represents the bandwidth parameter of the Gaussian kernel function. The Euclidean norm is represented. The candidate sentiment prototype cluster center with the smallest weighted distance is determined as the nearest sentiment prototype cluster center. In some embodiments, the number of intrinsic modes decomposed by empirical wavelet transform is between 3 and 5. In some embodiments, the dimension of the geometric morphology descriptor set is equal to the number of intrinsic modes multiplied by 3. It can be understood that the skewness, kurtosis, and fractal dimension of the envelope extreme point distribution describe the morphological characteristics of the intrinsic modes. It can be understood that the sparse autoencoder learns compressed representations through training. Optionally, the number of sentiment prototype cluster centers in the codebook is preset. Optionally, the cosine similarity threshold is adjusted according to the application scenario. In specific implementations, data comparison is reflected in the change of feature dimension before and after compression of the geometric morphology descriptor set. Before compression, the feature dimension is based on the number of intrinsic modes, and after compression, the feature dimension is fixed to a lower dimension. In the process of sentiment prototype cluster center matching, the cosine similarity threshold screening reduces the number of candidate sentiment prototype cluster centers, and the weighted distance calculation is only performed on the candidate sentiment prototype cluster centers.
[0023] Example 2: See Figure 3 The emotion evolution prediction module retrieves all occurrences of the nearest emotion prototype cluster center within a preset time window. Connecting the historical emotion prototype cluster centers corresponding to these time points forms the historical evolution trajectory of that emotion prototype cluster center in the emotion space. A time series prediction model is used to extrapolate this historical evolution trajectory, with an extrapolation step size of one time window to obtain the extrapolated coordinates of the emotion prototype cluster centers. The direction of the line connecting the extrapolated coordinates of the emotion prototype cluster centers and the current coordinates is defined as a possible evolution path. The dynamic emotion dictionary uses the emotion dimension as its root node, dividing it into multiple emotion category sub-nodes. Each emotion category sub-node stores specific emotion semantic tag leaf nodes. The direction vector of the possible evolution path in the emotion space is mapped to the corresponding emotion dimension in the dynamic emotion dictionary. The intensity parameter is determined based on the ratio of the magnitude of the direction vector to the average fluctuation amplitude of the emotion prototype cluster center on its historical evolution trajectory. In the emotion category sub-nodes of the mapped emotion dimension, emotion semantic tag leaf nodes within a pre-defined intensity range are matched according to the intensity parameter.
[0024] In practical implementation, the emotion evolution prediction module retrieves all time points within a preset historical window for the nearest emotion prototype cluster center. The historical window length is set to include multiple consecutive time windows. Connecting the historical emotion prototype cluster centers corresponding to the time points forms the historical evolution trajectory of the emotion prototype cluster centers in the emotion space. The historical evolution trajectory is composed of a series of coordinate points in the emotion space connected in chronological order. A time series prediction model is used to extrapolate the historical evolution trajectory, with an extrapolation step size of one time window, to obtain the extrapolated coordinates of the emotion prototype cluster centers. The direction of the line connecting the extrapolated coordinates of the emotion prototype cluster centers and the current coordinates of the emotion prototype cluster centers is defined as the possible evolution path. The extrapolated coordinates of the emotion prototype cluster centers can be calculated using the following formula: in: This represents the coordinates of the center of the predicted extrapolated sentiment archetype cluster. This represents the coordinates of the nearest cluster of sentiment archetypes corresponding to the current time window t. The coordinates of the center of the sentiment prototype cluster corresponding to the i-th preceding time window in the historical window are represented by , the i-th autoregressive coefficient is represented by , and k represents the order of the autoregressive model. The dynamic sentiment dictionary takes the sentiment dimension as the root node, and divides it into multiple sentiment category sub-nodes. Each sentiment category sub-node stores specific sentiment semantic label leaf nodes. The direction vectors of possible evolution paths in the sentiment space are mapped to the corresponding sentiment dimensions in the dynamic sentiment dictionary. The direction vectors are calculated based on the difference between the extrapolated coordinates of the sentiment prototype cluster center and the current coordinates of the sentiment prototype cluster center. The intensity parameter is determined based on the ratio of the magnitude of the direction vector to the average fluctuation amplitude of the sentiment prototype cluster center on the historical evolution trajectory. The intensity parameter is a scalar value. In the sentiment category sub-nodes under the mapped sentiment dimension, sentiment semantic label leaf nodes with pre-defined intensity intervals are matched according to the intensity parameter. Each sentiment semantic label leaf node is associated with an intensity interval.
[0025] Example 3: Basic motion units are defined by keyframe sequences and joint angle change curves. The virtual character motion library uses emotional semantic tags as the main index. Each emotional semantic tag is associated with a group of basic motion units sorted by intensity parameters. Based on the determined emotional semantic tags, the corresponding group of basic motion units is located in the virtual character motion library. Based on the intensity parameters, interpolation is performed in the group of basic motion units to select the two basic motion units closest to the intensity parameters as reference units. The keyframe sequences and joint angle change curves of the two reference units are linearly mixed to generate the final basic motion unit used for driving. The motion synthesis module determines the duration of each basic motion unit, adjusts the playback rate of the basic motion unit according to the time window length and motion rhythm parameters, extracts the end posture keyframe of the previous basic motion unit and the start posture keyframe of the next basic motion unit at the junction of two adjacent basic motion units, inserts a transition keyframe sequence at the junction. The transition keyframe sequence is generated by a posture interpolation algorithm so that the joint angle of the virtual character changes smoothly from the end posture keyframe to the start posture keyframe. The adjusted basic motion units and the inserted transition keyframe sequence are encoded into a timestamped motion command stream in chronological order.
[0026] In practice, basic motion units are defined by keyframe sequences and joint angle change curves. The keyframe sequence specifies the virtual character's posture at several specific time points during the action, and the joint angle change curves describe the angle values of each joint's continuous movement between keyframes. The virtual character motion library uses emotion semantic tags as the main index. Each emotion semantic tag is associated with a set of basic motion units sorted by intensity parameters. For example, the emotion semantic tag "pleasure_neutral" is associated with a set containing five basic motion units with intensity parameter values of 1.0, 1.25, 1.5, 1.75, and 2.0, respectively. The corresponding basic motion unit group is located in the virtual character motion library based on the determined emotion semantic tag. Interpolation is performed within the basic motion unit group based on the intensity parameters to select the two basic motion units closest to the intensity parameters as reference units. For example, when the intensity parameter is 1.7, two basic motion units with intensity parameters of 1.5 and 1.75 are selected as reference units from the basic motion unit group associated with "pleasure_neutral". The keyframe sequences and joint angle change curves of two reference units are linearly blended to generate the final base motion unit used for driving. The blending process is controlled by interpolation parameters, which are calculated based on the relative position of the intensity parameter 1.7 between the intensity parameters 1.5 and 1.75 of the two reference units. The formula for the keyframe pose of the final base motion unit used for driving is expressed as: in: This represents the keyframe pose matrix of the underlying action unit ultimately used for driving at a certain point in time. This represents the keyframe pose matrix of the first reference base action unit with the lower intensity parameter at the corresponding time point. This represents the keyframe pose matrix of the second reference base action unit with higher intensity parameters at the corresponding time point. This represents the interpolation parameters.
[0027] The motion synthesis module determines the duration of each basic motion unit and adjusts its playback rate based on the time window length and motion rhythm parameters, which are coefficients used to scale the original duration of the basic motion units. At the junction of two adjacent basic motion units, the ending pose keyframe of the previous basic motion unit and the starting pose keyframe of the next basic motion unit are extracted. A transition keyframe sequence is inserted at the junction. This transition keyframe sequence is generated using a pose interpolation algorithm to smoothly change the joint angles of the virtual character from the ending pose keyframe to the starting pose keyframe. The pose interpolation algorithm uses spherical linear interpolation to process the joint rotation data. The adjusted basic motion units and the inserted transition keyframe sequence are encoded in chronological order into a timestamped motion command stream. The timestamped motion command stream contains the timestamp of each keyframe and the corresponding full-body joint angle data of the virtual character.
[0028] In practical implementation, the example scenario involves the emotional semantic tag "pleasant_moderate" and an intensity parameter of 1.7. The data comparison is reflected in the selection and generation process of basic motion units. For example, in the "pleasant_moderate" basic motion unit group, the basic motion unit with an intensity parameter of 1.5 includes a "smiling and waving" action, in which the hand joint reaches its highest point in the middle frame of its keyframe sequence; the basic motion unit with an intensity parameter of 1.75 includes a "laughing and waving" action, in which the hand joint swings more and reaches a higher position in the middle frame. By linearly blending to generate the final basic motion unit used for driving, the height of its hand joint in the middle frame will be between the two reference basic motion units mentioned above. When splicing the motion, assuming that the ending posture keyframe of the previous final basic motion unit used for driving is a standing posture, and the starting posture keyframe of the next final basic motion unit used for driving is a half-squatting posture in preparation for sitting down, the transition keyframe sequence inserted at the junction will allow the virtual character to gradually and smoothly transition from a standing posture to a half-squatting posture.
[0029] See Figure 4This is a comparison chart of the blending effects of keyframe poses of a virtual character's motion units, used to demonstrate the linear blending results of basic motion units in emotional feedback interaction. After blending, the hand joint height of the motion unit always falls between the two reference units. The height of keyframe 2 (middle) is close to reference unit 2 (the higher-intensity motion), conforming to the interpolation logic that a target intensity of 1.7 is closer to 1.75. The heights of keyframes 1 and 3 balance the amplitude of the two reference units, demonstrating the precise control of motion intensity by linear blending. The intensity parameter of the reference unit is positively correlated with the motion amplitude (the higher the intensity, the greater the joint height). The amplitude of the blended motion dynamically matches the interpolation ratio with the keyframes, ensuring a natural transition of the motion. This chart verifies the effectiveness of the linear blending algorithm for motion units: by selecting reference actions of different intensities and interpolating, new motion units that meet the target emotional intensity can be generated, preserving the core pose features of the reference actions while accurately matching the intensity requirements of emotional semantics. This is the core technical support for motion synthesis in virtual character emotional feedback interaction.
[0030] Example 4: The feedback calculation module drives the skeletal model of the virtual character to move according to the action command flow and synchronously updates the pixel values of the regions related to the emotional semantic tags in the facial texture map of the virtual character. It captures the user's feedback bioelectric signals to the virtual character's movement in real time, calculates the deviation of the feedback bioelectric signals from the center of the currently driving emotional prototype cluster in the feature space, and corrects the matching weights of the emotional prototype mapping network online based on the deviation, and updates the historical evolution trajectory of the emotional prototype cluster center in the emotional space. The real-time captured user feedback bioelectric signals are subjected to the same adaptive segmentation and hierarchical dimensionality reduction compression processing as the original sequence of bioelectric signals from the user's facial region to obtain the feedback emotional feature vector. The Mahalanobis distance between the feedback emotional feature vector and the center of the currently driving emotional prototype cluster of the virtual character is calculated, and the Mahalanobis distance is standardized so that its value range falls between zero and one to obtain the standardized deviation.
[0031] In practice, the feedback calculation module drives the skeletal model of the virtual character to move according to the action command flow. The joint angles of the skeletal model are updated frame by frame based on the timestamp data in the action command flow, and the pixel values of the areas related to the emotional semantic label in the facial texture map of the virtual character are updated synchronously. For example, when the emotional semantic label is "pleasure", the pixel values of the corner of the mouth area in the facial texture map are adjusted to present a smiling texture. The bioelectrical signals of the user's feedback to the virtual character's movement are captured in real time. The feedback bioelectrical signals are collected by electrodes worn on the user's face. The deviation of the feedback bioelectrical signals from the center of the currently driving emotional prototype cluster in the feature space is calculated. Based on the deviation, the matching weights of the emotional prototype mapping network are corrected online, and the historical evolution trajectory of the emotional prototype cluster center in the emotional space is updated. The real-time captured user feedback bioelectrical signals are subjected to the same adaptive segmentation and hierarchical dimensionality reduction compression processing as the original sequence of bioelectrical signals from the user's facial region to obtain the feedback emotional feature vector. The adaptive segmentation uses the same time window length, and the hierarchical dimensionality reduction compression uses the same sparse autoencoder model. Calculate the Mahalanobis distance between the feedback emotion feature vector and the center of the emotion prototype cluster currently driving the virtual character. Standardize the Mahalanobis distance so that its value falls between zero and one to obtain the standardized deviation. The standardization formula is: Where: represents the standardization deviation, This represents the Mahalanobis distance between the feedback sentiment feature vector and the center of the current driving sentiment prototype cluster. This represents the adjustment parameters used to control the shape of the standardized curve. Take positive real numbers. The formula for calculating Mahalanobis distance is: in: This represents the feedback sentiment feature vector. This represents the coordinate vector of the center of the current driving emotion prototype cluster. The covariance matrix represents the set of historical sentiment feature vectors associated with the current driving sentiment prototype cluster center. Indicates matrix transpose. This represents the inverse of a matrix.
[0032] In the specific implementation, the example scenario involves a virtual character performing a waving gesture corresponding to the emotional semantic label "pleasure_neutral". After the user observes this gesture, facial bioelectrical signals are captured in real time. The feedback bioelectrical signals are processed to obtain feedback emotional feature vectors, and the coordinate vector of the center of the current driving emotional prototype cluster is [0.65, 0.85] in the emotional space. Data comparison is reflected in the Mahalanobis distance and standardized deviation values corresponding to different feedback emotional feature vectors. Refer to Table 1, which shows the Mahalanobis distance and standardized deviation calculated with three different feedback emotional feature vectors and the same center of the current driving emotional prototype cluster, where the adjustment parameter... Set to 1.0.
[0033] Table 1: Calculation of Mahalanobis Distance between Feedback Emotion Feature Vectors and Emotion Prototype Cluster Centers Feedback sentiment feature vector (f) Current driving emotion prototype cluster center (m) Mahalanobis distance (D) Standardized deviation (δ) [0.62,0.82] [0.65,0.85] 0.12 0.11 [0.45,0.70] [0.65,0.85] 0.85 0.46 [0.20,0.40] [0.65,0.85] 1.64 0.62 In some embodiments, adjusting parameters The value of is determined through offline calibration, the calibration process being based on the distribution range of Mahalanobis distances in historical interaction data. In some embodiments, the covariance matrix... The distance is calculated using the most recent one hundred historical emotional feature vectors corresponding to the center of the current driving emotional prototype cluster. It's understandable that Mahalanobis distance considers the correlation between different dimensions of the feature vectors. It's also understandable that the normalization process maps the Mahalanobis distance to a fixed interval for easier threshold comparison. Optionally, facial texture mapping updates are achieved by looking up a predefined texture mapping table, which associates emotional semantic labels with facial region pixel adjustment values. Optionally, the acquisition of feedback bioelectrical signals uses the same set of electrodes and sampling hardware as the initial raw bioelectrical signal sequence.
[0034] Example 5: Set a deviation threshold. When the standardized deviation exceeds the deviation threshold, trigger the online correction process. The feedback sentiment feature vector is used as a new data point and merged with the currently driven sentiment prototype cluster center and its historical evolution trajectory data. The centroid position of the sentiment prototype cluster center is recalculated using the merged data and the coordinates of the corresponding sentiment prototype cluster center in the sentiment prototype mapping network codebook are updated. Based on the updated sentiment prototype cluster center coordinates, the matching weight matrix of the relevant sentiment prototype cluster centers in the sentiment prototype mapping network is recalculated.
[0035] In practice, a deviation threshold is set, which is a preset constant between zero and one. When the standardized deviation exceeds the deviation threshold, an online correction process is triggered. The online correction process merges the feedback sentiment feature vector as a new data point with the currently driving sentiment prototype cluster center and its historical evolution trajectory data. This merging operation means adding the feedback sentiment feature vector to the historical dataset used to describe the currently driving sentiment prototype cluster center. The centroid position of the sentiment prototype cluster center is recalculated using the merged data. The centroid position is obtained by calculating the arithmetic mean of the coordinates of each dimension of all sentiment feature vectors in the historical dataset. The coordinates of the corresponding sentiment prototype cluster center in the sentiment prototype mapping network codebook are updated, replacing the stored coordinates with the newly calculated centroid position coordinates. Based on the updated sentiment prototype cluster center coordinates, the matching weight matrix of the relevant sentiment prototype cluster centers in the sentiment prototype mapping network is recalculated. The matching weight matrix is used to calculate the weighted distance between the input sentiment feature vector and each sentiment prototype cluster center in subsequent mapping processes. The update calculation of the matching weight matrix can be based on the formula: in: This represents the weight vector in the updated matching weight matrix that is related to the center of the current driving sentiment prototype cluster. This represents the weight vector in the matching weight matrix before the update. This represents the preset learning rate parameter. This indicates the standardized deviation that triggered this update. This represents the feedback sentiment feature vector. This represents the coordinate vector of the center of the emotion prototype cluster before the update. This represents the coordinate vector of the center of the updated emotion prototype cluster. This represents the transpose operation of a vector. It can be understood that the formula adjusts the weights by combining the standardized deviation, the difference between the feedback feature and the old centroid, and the centroid update amount.
[0036] In some embodiments, the deviation threshold is set to 0.5. In some embodiments, the learning rate parameter... The value is set to 0.01. Optionally, the size of the historical evolution trajectory dataset is capped; if adding new data points causes the dataset to exceed the cap, the oldest data points in the dataset will be removed. Optionally, the matching weight matrix is recalculated only if the standardized deviation exceeds the deviation threshold twice consecutively. In a specific implementation, the example scenario follows Example 4, assuming the standardized deviation... The calculated value is 0.62, and the set deviation threshold is 0.5. Since 0.62 > 0.5, the online correction process is triggered. Data comparison is reflected in the numerical changes before and after the update of the sentiment prototype cluster center coordinates and the matching weight matrix. The currently driving sentiment prototype cluster center coordinate vector... The range is [0.65, 0.85], and its historical dataset contains 100 historical sentiment feature vectors. The new feedback sentiment feature vector... The value is [0.20, 0.40]. The centroid coordinate vector is recalculated after being added to the historical dataset. The coordinates of the center of this emotion prototype cluster in the codebook are then updated from [0.65, 0.85] to [0.643, 0.847]. Assume the corresponding weight vector in the matching weight matrix before the update... The learning rate parameter is [0.2, -0.1]. The update amount is calculated based on the formula: This update is extremely small, reflecting that when the center point coordinates change slightly, the adjustment range of the weight vector is also correspondingly small. Updated weight vector The values are [0.19997768, -0.10002232]. It can be understood that the online correction process enables the system to fine-tune its internal emotional representations and mapping relationships based on real-time user feedback.
[0037] See Figure 5 This is a time-domain fluctuation plot of signal strength, used to show the intensity change of a certain type of dynamic signal (such as bioelectricity or sensor-acquired signals) over time. The signal strength exhibits high-frequency, irregular fluctuations throughout the observation interval, without obvious periodic or trend changes, indicating that the signal belongs to the category of random disturbances or non-steady-state dynamic signals. The signal strength remains consistently within the range of -2 to 2, without any extreme jumps outside this range, demonstrating the signal's amplitude stability, which can serve as a fundamental characteristic for subsequent signal processing. This plot is typically used for feature observation in the signal preprocessing stage: by analyzing the signal's fluctuation amplitude and frequency distribution, it can be determined whether preprocessing operations such as denoising and smoothing are necessary, and it can also serve as the raw data basis for subsequent feature extraction.
[0038] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A virtual character interaction device with emotional feedback function, characterized in that, The device includes: The signal segmentation module acquires the raw sequence of bioelectric signals from the user's facial region and adaptively segments the raw sequence of bioelectric signals into multiple signal segments using time windows as units. The hierarchical dimensionality reduction module performs hierarchical dimensionality reduction and compression on each signal segment to obtain the sentiment feature vector corresponding to each time window. The sentiment feature vector contains a set of geometric morphology descriptors for the signal segment. The emotion prototype mapping module inputs the emotion feature vector into a pre-trained emotion prototype mapping network, which matches the nearest emotion prototype cluster center based on the feature space distance. The emotion evolution prediction module predicts the possible evolution path of the center of the emotion prototype cluster in the next time window based on the historical evolution trajectory of the center of the emotion prototype cluster in the emotion space. Based on the possible evolution path, it retrieves the corresponding emotion semantic tags and intensity parameters from the dynamic emotion dictionary, which is organized in a tree structure. The motion synthesis module selects basic motion units from the virtual character motion library based on the emotional semantic tags and intensity parameters; it then splices and smoothly transitions the selected basic motion units in the time domain to generate a continuous motion instruction stream.
2. The virtual character interaction device with emotional feedback function according to claim 1, characterized in that, The step of performing hierarchical dimensionality reduction and compression on each signal segment to obtain the sentiment feature vector corresponding to each time window includes: Perform empirical wavelet transform on each signal segment to decompose the multiple intrinsic modes contained in the signal segment; Calculate the distribution of envelope extrema for each intrinsic mode, extract the skewness, kurtosis and fractal dimension of the envelope extrema distribution, and construct a set of geometric morphology descriptors for the signal segment. The set of geometric morphology descriptors is compressed and encoded using a sparse autoencoder to remove redundant information and retain the main feature patterns, outputting a low-dimensional sentiment feature vector.
3. A virtual character interaction device with emotional feedback function according to claim 2, characterized in that, The emotion prototype mapping network matches the nearest emotion prototype cluster center based on feature space distance, including: The emotional prototype mapping network maintains a codebook composed of multiple emotional prototype cluster centers formed by clustering historical emotional feature vectors; Calculate the cosine similarity between the input sentiment feature vector and the center of each sentiment prototype cluster in the codebook; Several emotion prototype cluster centers with cosine similarity exceeding a set threshold are selected as candidates, and the weighted distance between the input emotion feature vector and each candidate emotion prototype cluster center under Gaussian kernel function mapping is calculated. The candidate emotion prototype cluster center with the smallest weighted distance is determined as the nearest emotion prototype cluster center.
4. A virtual character interaction device with emotional feedback function according to claim 3, characterized in that, The step of predicting the possible evolution path of the emotional prototype cluster center in the next time window based on the historical evolution trajectory of the emotional prototype cluster center in the emotional space includes: Retrieve all time points in a historical window of a preset time length from the nearest center of the emotion prototype cluster; Connect the centers of historical emotional prototype clusters corresponding to the time points to form the historical evolution trajectory of the centers of these emotional prototype clusters in the emotional space; The historical evolution trajectory is extrapolated using a time series prediction model with an extrapolation step size of one time window to obtain the extrapolated emotional prototype cluster center coordinates. The direction of the line connecting the extrapolated emotional prototype cluster center coordinates and the current emotional prototype cluster center coordinates is defined as the possible evolution path.
5. A virtual character interaction device with emotional feedback function according to claim 4, characterized in that, The step of retrieving corresponding sentiment semantic tags and intensity parameters from the dynamic sentiment dictionary based on the possible evolution paths includes: The dynamic sentiment dictionary takes the sentiment dimension as the root node, and divides the sentiment dimension into multiple sentiment category sub-nodes. Each sentiment category sub-node stores specific sentiment semantic tag leaf nodes. Based on the direction vector of the possible evolution path in the emotion space, it is mapped to the corresponding emotion dimension in the dynamic emotion dictionary; The intensity parameter is determined based on the ratio of the magnitude of the directional vector to the average fluctuation amplitude of the center of the emotional prototype cluster along its historical evolution trajectory. In the sentiment category sub-nodes mapped to the sentiment dimension, sentiment semantic label leaf nodes with pre-defined intensity ranges are matched according to the intensity parameter.
6. A virtual character interaction device with emotional feedback function according to claim 5, characterized in that, The step of selecting basic action units from the virtual character action library based on the emotional semantic tags and intensity parameters includes: The basic motion unit is defined by a keyframe sequence and a joint angle change curve; The virtual character action library uses emotional semantic tags as the main index, and each emotional semantic tag is associated with a set of basic action units sorted by intensity parameters. Based on the determined emotional semantic tags, the corresponding basic action unit group is located in the virtual character action library; Interpolation is performed in the basic motion unit group based on the intensity parameters, and the two basic motion units that are closest to the intensity parameters are selected as reference units. The keyframe sequences and joint angle change curves of the two reference units are linearly blended to generate the final basic motion unit used for driving.
7. A virtual character interaction device with emotional feedback function according to claim 6, characterized in that, The step of splicing and smoothing the selected basic action units in the time domain to generate a continuous action instruction stream includes: Determine the duration of each basic motion unit, and adjust the playback rate of the basic motion unit according to the time window length and motion rhythm parameters; At the junction of two adjacent basic motion units, extract the end pose keyframe of the previous basic motion unit and the start pose keyframe of the next basic motion unit. A transition keyframe sequence is inserted at the junction. The transition keyframe sequence is generated by a pose interpolation algorithm so that the joint angle of the virtual character changes smoothly from the end pose keyframe to the start pose keyframe. The adjusted basic motion units and the inserted transition keyframe sequences are encoded in chronological order into a timestamped motion instruction stream.
8. A virtual character interaction device with emotional feedback function according to claim 1, characterized in that, The device also includes a feedback calculation module for performing: The motion command stream drives the skeletal model of the virtual character to move, and synchronously updates the pixel values of the areas related to the emotional semantic tags in the facial texture map of the virtual character. Real-time capture of user feedback bioelectric signals to virtual character movements, and calculation of the deviation of the feedback bioelectric signals from the center of the current driving emotion prototype cluster in the feature space; The matching weights of the emotional prototype mapping network are corrected online based on the deviation, and the historical evolution trajectory of the emotional prototype cluster center in the emotional space is updated.
9. A virtual character interaction device with emotional feedback function according to claim 8, characterized in that, The calculation of the deviation between the feedback bioelectric signal and the center of the current driving emotion prototype cluster in the feature space includes: The real-time captured user feedback bioelectric signals are subjected to the same adaptive segmentation and hierarchical dimensionality reduction and compression processing as the original sequence of bioelectric signals from the user's facial region to obtain the feedback emotion feature vector. Calculate the Mahalanobis distance between the feedback emotion feature vector and the center of the emotion prototype cluster currently driving the virtual character; The Mahalanobis distance is standardized so that its value range falls between zero and one, thus obtaining the standardized deviation.
10. A virtual character interaction device with emotional feedback function according to claim 9, characterized in that, The online correction of the matching weights of the emotion prototype mapping network based on the deviation includes: Set a deviation threshold; when the standardized deviation exceeds the deviation threshold, trigger an online correction process. The feedback emotion feature vector is used as a new data point and merged with the current driving emotion prototype cluster center and its historical evolution trajectory data; The centroid position of the emotional prototype cluster center is recalculated using the merged data, and the coordinates of the corresponding emotional prototype cluster center in the emotional prototype mapping network codebook are updated. Based on the updated coordinates of the emotion prototype cluster centers, the matching weight matrix of the relevant emotion prototype cluster centers in the emotion prototype mapping network is recalculated.