A semantic driving-based robot facial expression real-time generation method and system

Abstract

The application relates to a kind of real-time generation method and system of robot facial expression based on semantic driving, belong to intelligent robot interaction technical field.Wherein, the method comprises: obtaining user voice interaction data;Continuous modeling is carried out on the change process of robot facial expression, and the semantic contribution weight of different emotional dimension is obtained;Converge robot short-time expression mutation, output corresponding target expression driving signal;Obtain expression adaptation score, and determine the matching degree of current robot expression and user semantics in combination with the expression adaptation score;When the matching degree reaches preset expression switching threshold, the current robot expression is used as transition reference expression to carry out expression smooth migration, and the target expression driving signal is compensated in combination with semantic emotional inertia, and the corresponding robot facial expression is updated in real time.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot interaction technology, specifically relating to a method and system for real-time generation of robot facial expressions based on semantics-driven principles. Background Technology

[0002] With the rapid development of artificial intelligence, natural language processing, and robot control technologies, intelligent robots are increasingly being applied in various fields such as intelligent customer service, family companionship, educational assistance, and public services. Compared to traditional robots that rely solely on voice output or text interaction, robots with human-like facial expressions can enhance emotional feedback during interactions through facial expression changes, improving the naturalness and immersion of user-robot interaction. Therefore, how to generate robot facial expressions in real time that match the interactive context based on user input has become an important research direction in the field of intelligent robots.

[0003] Existing robotic facial expression generation technologies typically rely on pre-defined emotion tags or fixed mapping rules. This involves classifying user speech or text information into emotion categories to determine the corresponding target emotion category and then generating the appropriate expression using pre-configured facial motion parameters. However, these methods primarily focus on emotion matching within a single interaction, lacking continuous analysis of contextual semantic relationships. They fail to fully understand the emotional changes of users during continuous dialogue, leading to discrepancies between the robot's facial expression responses and the actual interactive context. For example, even as a user's emotions gradually change, the robot may still directly switch to the target expression based on the current sentence's semantics, resulting in overly rapid expression changes and unnatural emotional transitions.

[0004] Furthermore, existing expression-driven methods typically employ direct control of the target expression. When the current expression state differs significantly from the target expression state, abrupt changes in facial motion parameters can easily occur, causing jumps in the robot's facial expressions and reducing the continuity and human-likeness of the robot's expression display. Simultaneously, due to the lack of effective utilization of historical interaction states, the robot cannot adjust the intensity of its current expression response based on the emotional accumulation during continuous dialogue, resulting in significant short-term fluctuations in expression and insufficient emotional continuity.

[0005] On the other hand, some existing technologies analyze user emotions by introducing emotion recognition models, but these typically rely solely on current speech, text, or single emotional features for judgment. They fail to consider the dynamic impact of different semantic and emotional dimensions on the facial expression change process, nor do they establish a correlation model between semantic information and the facial expression transfer process. This results in the generated robot expressions failing to accurately reflect emotional states in complex interactive scenarios. Furthermore, during actual robot operation, factors such as environmental changes, semantic understanding errors, and differences in the response of expression execution mechanisms can lead to discrepancies between the target expression and the actual displayed expression. Existing methods lack effective expression matching verification and feedback correction mechanisms.

[0006] Therefore, there is a need for a real-time robot facial expression generation method that can combine contextual semantic information to continuously model the process of robot facial expression changes and dynamically adjust based on semantic emotion change trends, expression migration status, and feedback matching results, in order to improve the continuity, accuracy, and natural interaction capabilities of robot facial expression responses. Summary of the Invention

[0007] To address the aforementioned problems in the existing technology, this invention provides a semantically driven real-time facial expression generation method for robots. The objective of this invention can be achieved through the following technical solutions: Acquire user voice interaction data; Extract contextual semantic information from the user's voice interaction data; continuously model the robot's facial expression change process based on the contextual semantic information to obtain the semantic contribution weights of different emotional dimensions; Based on the semantic contribution weight, the response transition state in the robot's facial expression transition is constrained, and the short-term facial expression change of the robot is converged by combining the constrained response transition state, and the corresponding target facial expression driving signal is output. Based on the deviation between the current robot expression and the expected robot expression, an expression adaptation score is obtained by comparing the target expression driving signal. This expression adaptation score is then used to determine the degree of matching between the current robot expression and the user's semantics. When the matching degree reaches the preset expression switching threshold, the current robot expression is used as the transition benchmark expression for smooth expression migration, and the target expression driving signal is combined with semantic emotion inertia compensation to update the corresponding robot facial expression in real time.

[0008] Specifically, the process of capturing contextual semantic information is as follows: The user's voice interaction data is converted into a corresponding voice-text sequence; Based on the speech-text sequence, the semantic interaction relationship in the current dialogue scenario is matched to obtain a semantic representation vector. The current user interaction intent is determined by combining the semantic representation vector, and contextual semantic information is generated. The contextual semantic information includes: speech text semantic features, dialogue context association information, user interaction intent information, emotional tendency information, semantic context information, and semantic temporal information, which are used to characterize the semantic content, emotional state, and continuous change trend of the user in the current interaction process, and provide semantic driving basis for robot facial expression generation.

[0009] Specifically, the process of continuously modeling the changes in the robot's facial expressions is as follows: Extract the initial facial expression state parameters corresponding to different emotional dimensions; The initial facial expression state parameters include: facial expression posture parameters and emotional dimension parameters, which are used to describe the spatial state, emotional state and dynamic change characteristics of the robot's current facial expression, and provide a state basis for subsequent continuous modeling of facial expression changes and calculation of semantic contribution weights.

[0010] The initial facial expression state parameters are mapped to a preset emotional dimension space, the current displayed facial expression state is associated with the historical displayed facial expression state, and the dynamic influence weight of different semantic emotional dimensions on the change process of facial expression display parameters is determined according to the response state during the facial expression display interaction process, so as to obtain the corresponding semantic contribution weight. The preset emotional dimension space is used to: map the robot's facial expression state parameters to a unified emotional feature representation space, quantify and describe different emotional categories, emotional intensities, and emotional change directions, and establish the correlation between semantic emotional information and facial expression change parameters, so as to dynamically determine the degree of influence of different emotional dimensions on the robot's facial expression change process based on the user's context semantics.

[0011] Specifically, the process of constraining the transition state of the robot's facial expression using the semantic contribution weight is as follows: The facial expression display parameters corresponding to different emotional dimensions are dynamically weighted according to the semantic contribution weight to determine the degree of influence of each emotional dimension on the facial expression transition process, and the transition state adjustment parameters are generated according to the degree of influence. The transition state adjustment parameters include: expression migration speed adjustment parameters, display frame smoothing parameters, emotional response intensity adjustment parameters, expression trajectory correction parameters, and transition duration parameters; Based on the transition state adjustment parameters, the display frame change differences and expression migration speed in the response transition state are adjusted, and the expression migration trajectory is corrected according to the semantics of the expression display state to constrain non-semantic related expression mutation behavior.

[0012] Specifically, the process of comparing the deviation between the current robot expression and the expected robot expression is as follows: When the deviation between the robot's expression and the expected robot expression meets a preset condition, the expression matching confidence score between the current robot expression and the expected robot expression is obtained, and an expression adaptation score is generated based on the expression matching confidence score; when the deviation between the robot's expression and the expected robot expression does not meet the preset condition, the image deviation between the currently displayed expression frame and the target displayed expression frame is obtained, and the robot expression display status is updated based on the deviation.

[0013] Specifically, the process of combining semantic emotion inertial compensation with the target facial expression driving signal is as follows: Obtain the historical semantic sentiment state sequence and the current semantic sentiment state parameters; The semantic emotion state parameters include: semantic emotion category parameters, emotion response intensity parameters, emotion change trend parameters, semantic emotion association parameters, emotion persistence parameters, emotion fluctuation parameters, and emotion response offset parameters. Based on the historical semantic emotion state sequence, semantic inertia influence parameters are determined in the expression response process; based on the semantic emotion state parameters, the emotional response intensity corresponding to the current interaction semantic is mapped to generate target expression-driven correction parameters; and the continuous expression display state is reconstructed by combining the semantic inertia influence parameters and the target expression-driven correction parameters, and the corresponding robot facial expressions are updated in real time.

[0014] Specifically, the process of the convergence robot's short-term facial expression change is as follows: Obtain the facial expression response status between adjacent frames; Based on the expression response state, the abrupt change interval in the expression response state is determined. The expression jump amplitude between adjacent display frames within the abrupt change interval is converged with the current display frame as the center. The converged smooth sequence is then mapped to continuous expression display parameters, and the target expression driving signal is output.

[0015] Specifically, when the matching degree does not reach the preset expression switching threshold: Obtain facial expression deviation feedback parameters; The expression deviation feedback parameters include: expression state offset parameters, facial movement deviation parameters, expression feature difference parameters, emotional response deviation parameters, expression transfer error parameters, expression matching confidence parameters, and expression adjustment direction parameters; The semantic contribution weights are reallocated based on the expression deviation feedback parameters, and the emotional dimension response ratio corresponding to the target expression state is corrected based on the allocated semantic contribution weights to generate robot expression transition control parameters. The robot's facial expression transition control parameters include: facial expression transition amplitude parameters, facial expression response speed parameters, inter-frame change constraint parameters, transition stability parameters, and facial expression retention time parameters.

[0016] Specifically, the process of smoothly transferring the current robot expression as a transitional baseline expression is as follows: Based on the target expression driving signal, the target facial motion parameters are determined, and a continuous migration trajectory between the initial state and the target state is established. The initial state of the facial expression migration is confirmed based on the continuous migration trajectory, and the continuity of the inter-frame transition in the initial state of the facial expression migration is constrained. Smooth migration control parameters are generated to execute the continuous transition from the current facial expression state to the target facial expression state.

[0017] Specifically, a semantically driven real-time facial expression generation system for robots includes: The voice acquisition unit is used to acquire user voice interaction data; A semantic parsing unit is used to extract contextual semantic information from the user's voice interaction data; based on the contextual semantic information, the robot's facial expression change process is continuously modeled to obtain the semantic contribution weights of different emotional dimensions; The expression-driven signal generation unit constrains the response transition state in the robot's expression transition based on the semantic contribution weight, and converges the robot's short-term expression change according to the constrained response transition state, and outputs the corresponding target expression-driven signal. The expression matching and verification unit compares the deviation between the current robot expression and the expected robot expression based on the target expression driving signal to obtain an expression adaptation score, and combines the expression adaptation score to determine the degree of matching between the current robot expression and the user's semantics. When the matching degree reaches the preset expression switching threshold, the current robot expression is used as the transition benchmark expression for smooth expression migration, and the target expression driving signal is combined with semantic emotion inertia compensation to update the corresponding robot facial expression in real time.

[0018] The beneficial effects of this invention are as follows: This invention provides a semantically driven method and system for real-time generation of robot facial expressions. By acquiring user voice interaction data and extracting contextual semantic information, the system continuously models the process of robot facial expression changes. It can dynamically analyze user interaction intentions by combining the semantic contribution weights corresponding to different emotional dimensions, transforming the robot expression generation process from single emotion matching to continuous emotional response based on semantic association, thereby improving the matching accuracy between robot expression and user semantic content.

[0019] By constraining the response transition state during the robot's facial expression transition process based on semantic contribution weights, and by adjusting the amplitude of facial expression changes, migration speed, and inter-frame change relationships corresponding to different emotional dimensions, the sudden changes generated during the target facial expression switching process can be effectively reduced, making the robot's facial expression changes smoother and more continuous, and improving the naturalness and anthropomorphism of the facial expression display.

[0020] By acquiring historical semantic and emotional state sequences and current semantic and emotional state parameters, semantic inertia influence parameters in continuous interaction processes are extracted, and the facial expression display state is reconstructed by combining target facial expression-driven correction parameters. This enables the robot to retain the emotional continuity characteristics in continuous dialogue processes, avoids excessive fluctuations in facial expression responses due to single semantic changes, and improves the stability of the robot's emotional expression.

[0021] Furthermore, the current robot expression is matched and verified against the expected robot expression. Based on the expression adaptation score feedback, the semantic contribution weight and the target expression state response ratio are adjusted, achieving closed-loop optimization control in the expression generation process. This reduces the impact of semantic understanding errors and execution deviations on the expression effect. Therefore, this invention can improve the accuracy, continuity, and intelligence level of robot expression responses in human-computer interaction scenarios. Attached Figure Description

[0022] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0023] Figure 1 This is a schematic diagram of the framework of a semantically driven real-time facial expression generation system for robots according to the present invention.

[0024] Figure 2 This is a schematic diagram illustrating the process of updating the display state of robot facial expressions in a semantically driven real-time robot facial expression generation system according to the present invention. Detailed Implementation

[0025] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0026] Please see Figure 1 This invention also provides a semantically driven real-time facial expression generation system for robots, specifically including: The voice acquisition unit is used to acquire user voice interaction data; A semantic parsing unit is used to extract contextual semantic information from the user's voice interaction data; based on the contextual semantic information, the robot's facial expression change process is continuously modeled to obtain the semantic contribution weights of different emotional dimensions; The expression-driven signal generation unit constrains the response transition state in the robot's expression transition based on the semantic contribution weight, and converges the robot's short-term expression change according to the constrained response transition state, and outputs the corresponding target expression-driven signal. The expression matching and verification unit compares the deviation between the current robot expression and the expected robot expression based on the target expression driving signal to obtain an expression adaptation score, and combines the expression adaptation score to determine the degree of matching between the current robot expression and the user's semantics. When the matching degree reaches the preset expression switching threshold, the current robot expression is used as the transition benchmark expression for smooth expression migration, and the target expression driving signal is combined with semantic emotion inertia compensation to update the corresponding robot facial expression in real time.

[0027] Example 1: System Overall Architecture and Execution Flow; Acquire user voice interaction data; The system uses a multi-microphone array deployed at the front of the robot to collect high signal-to-noise ratio user voice signals in real time and preprocess them (noise reduction, echo cancellation and endpoint detection) to form a continuous voice interaction data stream.

[0028] Extract contextual semantic information and perform continuous modeling; Deep semantic feature extraction: Voice interaction data is input into a real-time speech recognition (ASR) engine to generate corresponding text sequences. The system relies on a pre-trained contextual language model, combined with dialogue history profiles, to analyze deep semantic interaction relationships in the current dialogue scenario, mapping and generating high-dimensional semantic representation vectors. Based on these vectors, the system accurately identifies the user's current interaction intent (such as speaking, questioning, teasing, or giving instructions) and implicit emotional tendencies, thereby constructing semantic information with contextual relevance.

[0029] Multidimensional Emotional Continuous Modeling: A multidimensional emotional space (e.g., valence-arousal-control space) is used to model the continuous trajectory of the robot's facial expression changes. The system extracts initial facial features (such as eyebrow tilt, eye opening / closing, and lip curvature) under different emotional dimensions and maps these parameters into the emotional space. By establishing a temporal correlation between the current and historical facial expression display states, the dynamic influence of semantic emotion on the evolution trajectory of facial expression parameters during continuous dialogue is analyzed. Finally, the semantic contribution weights of different emotional dimensions to facial expression generation under the current semantic context are calculated.

[0030] Constrain the transition and convergence of facial expression changes in response, and output the target facial expression driving signal; Response Transition State Constraints: Based on the calculated semantic contribution weights, the facial muscle action parameters corresponding to each emotional dimension are dynamically weighted to accurately assess the degree of control each emotional dimension has over the smooth transition process of facial expressions, and generate transition state adjustment parameters accordingly. The system uses these adjustment parameters to dynamically fine-tune the inter-frame change gradient and the overall facial expression migration rate during the response transition phase, while also correcting the facial expression migration trajectory in conjunction with semantic logic, eliminating abrupt facial expression changes that are irrelevant to the current semantics from the source.

[0031] Short-term facial expression abrupt change convergence: Real-time monitoring of facial expression response status between adjacent display frames accurately locates abrupt change intervals caused by speech noise or semantic transient jitter. Using the currently displayed facial expression frame as the central kernel, an adaptive smoothing operator is employed to suppress the abrupt change amplitude of adjacent frames within the abrupt change interval, smoothing the drastically fluctuating discrete features into continuous facial muscle movement trajectories, and generating the final target facial expression driving signal.

[0032] Semantic matching degree is determined based on facial expression adaptation score; Deviation comparison and score generation: The system compares the expected expression state parsed from the target expression driving signal with the robot's current actual facial expression state in real time and calculates the deviation value between the two states.

[0033] When the preset conditions are met: when the deviation value is within the reasonable threshold range set by the system, the system calculates the physical consistency and semantic fit between the current expression and the expected expression, generates the expression matching confidence score, and converts it into a specific expression adaptation score.

[0034] When the preset conditions are not met: If the deviation exceeds the preset conditions (such as external physical intervention or hardware response lag), the system directly calculates the image pixel-level or kinematic deviation between the current display frame and the target display frame, and quickly updates the robot's expression display status to forcibly track the target.

[0035] Matching decision mechanism: By combining the expression adaptation score and the contextual semantic persistence, the overall matching degree between the current robot expression and the user's semantics is determined.

[0036] Smooth facial expression transfer and semantic-emotional inertia compensation; When the overall matching degree reaches the preset emoji switching threshold, the system triggers the emoji update mechanism: Baseline Smooth Transition: Using the robot's current real facial expression state as the transition reference point, target facial motion unit (AU) parameters are extracted based on the target facial expression driving signal to construct a continuous transition trajectory from the reference state to the target state. In the initial stage of the transition, inter-frame transition continuity constraints are applied to generate smooth transition control parameters, driving the facial servos or virtual mesh to perform a seamless transition.

[0037] Semantic Emotional Inertia Compensation: To prevent overly mechanical expression switching, the system introduces emotional inertia adjustment. Historical semantic emotional state sequences are extracted, and the sedimentation and continuation effects of emotions over time (i.e., semantic inertia parameters) are calculated. Simultaneously, parameters are corrected based on the current semantic emotional response intensity. The inertia impact parameters and correction parameters are superimposed and reconstructed to achieve real-time updates of anthropomorphic expressions that evoke lingering emotions.

[0038] In some embodiments, the specific process of matching semantic interaction relationships in the current dialogue scenario is further as follows: After receiving the speech-text sequence, the system completes the matching and representation of semantic interaction relationships in the dialogue scenario through the following stages: Phase 1: Multi-granularity feature segmentation and window construction; The system performs three-level segmentation on the real-time identified text sequences: lexical, syntactic, and discourse levels. Simultaneously, the system maintains a dynamic sliding window that automatically retrieves historical interaction text from the previous six rounds of the current dialogue (an engineering recommendation, which can be dynamically adjusted between 5 and 8 rounds depending on computing power, corresponding to a time span of approximately 15 to 30 seconds). This sliding window is used to capture long-distance semantic dependencies, preventing misjudgments of facial expressions caused by the literal meaning of single sentences.

[0039] Phase Two: Scene Semantic Matrix Matching; The system inputs the current sentence features and historical features within the sliding window into a preset "scene semantic interaction matrix." This matrix includes, but is not limited to, the following four core interaction dimensions: Interaction role dimension: Determine whether the user is currently in the role of "leading questioner", "passive listener", "emotional venting person" or "casual chatter participant".

[0040] Emotional transfer tendency dimension: Analyze the evolution trend of contextual emotions (e.g., from "neutral-questioning" to "anxious-dissatisfied", or from "calm" to "pleasure").

[0041] Urgency of Intent Dimension: Assess the immediacy of user needs (such as emergency help requests versus casual daily chat).

[0042] Contextual certainty dimension: assess whether the user's expression contains metaphors, irony, or semantic ambiguity.

[0043] Phase 3: Generation of high-dimensional semantic representation vectors; By cross-referencing the semantic interaction matrix, the system outputs a dense semantic representation vector of 768 dimensions (or 512 dimensions based on a lightweight model). This vector not only contains the literal meaning of the current sentence but also deeply integrates the interaction roles and emotional evolution trends. The system further extracts explicit intentions (such as "requesting comfort") and implicit emotional intensity (such as "mild sadness, intensity level 0.35") from this vector, ultimately synthesizing complete contextual semantic information.

[0044] In some embodiments, the specific process of associating the currently displayed expression state with the historically displayed expression state is as follows: Feature vectorization extraction of explicit facial expression image states; The system extracts the feature states of the currently displayed facial expression image (or dynamic facial expression frame sequence) in real time. These feature states include, but are not limited to: the emotional dimension scalar value corresponding to the facial expression image (such as the level classification of joy, surprise, and confusion), and the morphological features of key facial organs (such as pixel-level or key point-level normalized parameters such as the degree of eye opening and closing and the curvature of the mouth in the image), and establishes a feature vector representing the current displayed facial expression state.

[0045] Construct a temporal memory queue for image display; The system maintains a fixed-length emoji image display history memory queue in a memory buffer, recording in real time the emoji image numbers and their corresponding feature vectors that have been displayed on the screen in the past 5 to 15 times (corresponding to the past 1.0 to 3.0 seconds of interaction).

[0046] Image switching frequency extraction: The switching frequency and dwell time of emoji images over a period of time are calculated using this queue to assess the stability of the current screen display; Emotional evolution trajectory extraction: This cohort is used to analyze the direction of emotional progression contained in the historical image sequence (e.g., the evolution from "calm images" to "slightly confused images").

[0047] The feature vectors of newly displayed images are pushed into the queue, and historical nodes that exceed the queue length are automatically discarded, forming a continuous chain of historical image display states.

[0048] Introduce an emotional damping and inhibition inheritance mechanism for image switching; In the context of direct image switching, directly cutting images based on new semantics would cause "face-like flickering" on the screen. Therefore, the system introduces a switching damping mechanism based on historical image states: Temporal dwell time constraint: If the dwell time of the current image in the historical memory queue has not reached the preset "minimum visual dwell time threshold" (e.g., 300 milliseconds), the system automatically increases the inheritance weight of the current historical image features to prevent the instantaneous overwriting of new emoji images and prevent visual flickering; Emotional Ladder Inheritance: When images of the same emotional category are displayed consecutively in the historical memory queue, the system assigns a certain damping weight to the historical emotions when calculating new image requirements. Even if the current semantics undergo a drastic shift, the system will prioritize selecting intermediate transitional images that are "between historical expression images and target expression images" (e.g., instead of directly switching from "crying image" to "laughing image," it will prioritize inheriting the historical state and switching to "smiling through tears / slightly sobbing image").

[0049] Dynamic influence on weight and image retrieval index calculation; The system will map the display status of historical images, which integrate switching damping and stepped inheritance characteristics, onto a preset multi-dimensional emotional space: Matching degree and influence weight comparison: Evaluate the deviation between the sentiment state of historically displayed images and the current semantic sentiment target, and calculate the dynamic influence weight (i.e. semantic contribution weight) of different sentiment dimensions on image update decisions. Image transition control instructions are generated: Based on the calculated semantic contribution weight and the characteristics of historically displayed images, the optimal target expression image index or intermediate transition image sequence is calculated and retrieved from the preset expression image library to ensure that the replacement of screen images not only matches the current semantics but also has a continuous and gradual sense of change that conforms to human visual habits.

[0050] In this embodiment, as Figure 2 As shown, the specific process of comparing the deviation between the current robot expression and the expected robot expression is as follows: When the deviation between the robot's expression and the expected robot expression meets a preset condition, the expression matching confidence score between the current robot expression and the expected robot expression is obtained, and an expression adaptation score is generated based on the expression matching confidence score; when the deviation between the robot's expression and the expected robot expression does not meet the preset condition, the image deviation between the currently displayed expression frame and the target displayed expression frame is obtained, and the robot expression display status is updated based on the deviation.

[0051] In some embodiments, the specific process of dynamically weighting the facial expression display parameters corresponding to different emotional dimensions according to the semantic contribution weight is as follows: Dimensional decomposition and mapping of facial expression display parameters; The system pre-calibrates multi-dimensional expression display parameters for each image (or animated expression frame) in the expression image library. These expression display parameters mainly include: Emotional dimension parameters: such as scalar levels of joy, sadness, surprise, anger, and confusion, ranging from 0 to 100; Visual morphological feature parameters: such as the level of eye opening and closing curvature, the level of lip curvature, the level of eyebrow tilt shape, and other normalized visual scalars.

[0052] Weighted fusion of multidimensional parameters and generation of comprehensive target vector; The system dynamically weights and calculates the facial expression display parameters corresponding to different preset emotional dimensions based on the calculated semantic contribution weights. Amplifying dominant sentiment parameters: The sentiment dimension assigned a higher semantic contribution weight has its corresponding facial expression display parameters given a higher product coefficient in the weighted calculation, making it dominant in the final image selection decision; Secondary sentiment parameters are preserved: Secondary sentiment dimensions (such as subtle surprise or embarrassment) that are assigned lower semantic contribution weights have their corresponding display parameters superimposed on the comprehensive features at a lower ratio, which is used to fine-tune the visual details of the final image.

[0053] Transition state adjustment parameters are generated based on weighted parameters; After generating a comprehensive facial expression target feature vector, the system evaluates the emotional span and the degree of change between the current expression and historical expressions based on the weighted results, and generates transition state adjustment parameters: When the weighted composite features show a large emotional range (such as a sudden change from high sadness to high joy), the generated transition state adjustment parameters will reduce the image switching rate and instruct the system to search the image library and insert an intermediate expression image with "neutral / transitional" features. When the weighted composite features show a smooth change in emotion, the transition state adjustment parameter instructs the system to directly and smoothly replace it with the most matching target expression image.

[0054] In some embodiments, the specific process of mapping the emotional response intensity corresponding to the current interaction semantics is as follows: Multidimensional extraction of semantic and emotional tension; After acquiring the current interaction semantic information, the system extracts the emotional response intensity features of the current semantics from the following three dimensions, and generates a normalized emotional intensity scalar value (for example, setting the value range to 0 to 100): Lexical modification dimension: Identify degree adverbs (such as “very”, “slightly”, “extremely”), interjections, and punctuation combinations (such as “!?”) in the text to assess the amplification or reduction of underlying sentiment; Semantic expression of mood dimension: Analyzing the reinforcing effect of sentence structure (such as imperative sentences, rhetorical questions, and strong exclamations) on emotional expression; Acoustic feature auxiliary dimension (if speech stream is included): Extract the volume peak, speech rate variation and fundamental frequency (pitch) fluctuation of the user's speech as auxiliary correction factors for semantic sentiment intensity.

[0055] A hierarchical mapping of the emoji image library; The system pre-defines the facial expression images in the database according to the emotional intensity level (for example, dividing "joy" images into: Level 1 smiling / polite smile, Level 2 knowing smile, Level 3 exaggerated ecstasy / exuberant smile).

[0056] The system establishes a dynamic mapping range between emotional response intensity and image level: Low-intensity mapping: When the semantic emotion intensity scalar is in a low range (e.g., 0~30), the system maps and outputs low-tension static or slightly animated facial expression images to convey a convergent and polite interactive visual. High-intensity mapping: When the semantic emotion intensity scalar is in the high range (e.g., 71~100), the system maps and outputs high-tension, dynamic special effects or exaggerated facial deformation expression images (e.g., images / GIFs with spinning, flashing or violent body movements) to achieve strong emotional resonance.

[0057] The mapping results correct the target facial expression driving signal; The calculated emotional response intensity not only determines the selection of images, but is also used to reconstruct and correct the control parameters in the target expression driving signal: Display timeliness correction: The higher the intensity of the emotional response, the greater the increase in the minimum continuous display time of the mapped image (for example, a high-intensity angry image is forced to remain displayed for a longer time), ensuring that strong emotional images are not instantly washed away by subsequent weak semantics; Intermediate transition frame control: For high-intensity emotion mapping, the system dynamically inserts 1-2 medium-intensity transition guidance images between the current image and the target high-intensity image, so that the "explosive" strong response of the screen expression has a surging process that conforms to visual physiology.

[0058] In some embodiments, the specific process of mapping the converged smooth sequence to continuous facial expression display parameters is as follows: Analysis of the temporal features of smooth sequences; The system extracts the smoothed sequence after mutation convergence processing and analyzes the core control parameters of the sequence on a continuous time axis, including: Target image index features: a unique identifier number indicating the current emoticon image (or animation frame group) to be displayed in the sequence; Temporal transition span: The allowed transition time window (e.g., 150 milliseconds to 300 milliseconds) calculated in the sequence for switching from the currently displayed image to the target displayed image. Morphological evolution trend: The gradient magnitude of changes in key visual morphology (such as eye opening and closing, mouth curvature) between adjacent images recorded in the sequence.

[0059] Generate multi-dimensional continuous facial expression display parameters; The system maps and reconstructs the extracted temporal features in real time to generate continuous facial expression display parameters that directly affect the screen display rendering engine. These continuous facial expression display parameters specifically include at least one of the following control dimensions: Image transparency blending (Alpha blending) control parameters: map the parameters of the dual-channel transparency evolution curve between the currently displayed image and the target displayed image (e.g., the transparency of the current image gradually changes from 100% to 0%, and the transparency of the target image gradually changes from 0% to 100%), and control the screen to perform image overlap cross-fade in and out within the transition time window. Keyframe interpolation rendering parameters: When the target expression is a dynamic sequence of images, the intermediate rendering interpolation frame morphology parameters are calculated based on the morphological evolution trend and generated between two static images, instructing the rendering engine to draw the intermediate state image. Display frame rate and refresh rate adjustment parameters: dynamically adjust the rendering engine's drawing frame rate according to the urgency of the transition time window (e.g., dynamically increase from 15FPS in static display to 60FPS in the transition phase) to ensure a smooth screen transition process.

[0060] Parameter output and screen rendering execution; The system sends the generated continuous facial expression display parameters to the screen display driver module at a fixed rendering time sequence (e.g., outputting an update command every 16.6 milliseconds). The display driver module, according to the transparency change gradient, interpolated frame data, or frame rate control indicated by the parameters, completes a seamless visual transition from the current facial expression image to the target facial expression image within a specified time window, ultimately outputting a visually continuous and flicker-free robot facial expression.

[0061] In some embodiments, the specific process of constraining the continuity of the display inter-frame transition in the initial state of the facial expression transition is as follows: Temporal locking and feature extraction of the initial state during migration; After determining to initiate facial expression migration, the system locks the initial time point of the switch and the subsequent 30-60 milliseconds (corresponding to the first 2-3 frames of the rendering engine) as the initial state of the facial expression migration. The system extracts the following from this initial state: Current rendering status of the baseline image: including the real-time display brightness, contrast, and transparency of the current baseline emoji image on the screen; Initial morphological differences of the target image: Calculate the magnitude of morphological changes in the core areas such as the eyes and mouth between the target expression image and the current baseline expression image (e.g., the visual pixel difference between the mouth shape changing from closed to open).

[0062] Generate initial transition continuity constraint parameters; Based on the calculated magnitude of morphological abrupt changes, the system applies the following triple continuity constraints to the transition continuity between display frames within the initial migration state, generating smooth control parameters: Initial transparency gradient slope constraint (Alpha easing constraint): Limit the fade-in transparency of the target expression image in the initial 1st frame (for example, forcibly constrain the transparency of the target image in the initial 1st frame to a very low ratio of 5%~10%, instead of directly jumping to high transparency), to achieve "slow start (Ease-in)" at the beginning of the screen. Key shape edge smoothing constraint: If the pixel difference between the core organ region (such as the corner of the eye and the corner of the mouth) of the current image and the target image is too large, the system applies local Gaussian blur or edge softening constraint in the initial frame to eliminate the hard edge visual silhouette at the moment of the start of the image. Inter-frame change rate limit: Set the maximum color and brightness change threshold between adjacent display frames in the initial state. When the visual color jump between the initial frames exceeds the threshold, the change rate of the RGB color channels of the initial frame is automatically damped and truncated.

[0063] Smooth migration execution after constraints; The system sends the smooth transition control parameters generated by the above constraints to the screen rendering engine. The rendering engine completes the rendering of the initial stage of the expression transition according to the logic of "low starting transparency fade-edge softening-progressive acceleration (S-curve start)", and then smoothly connects to the subsequent normal transition trajectory, driving the screen to complete the seamless transition from the current base expression to the target expression.

[0064] Example 2: Closed-loop feedback and anomaly handling in heterogeneous scenarios; If the calculated overall matching degree does not reach the preset expression switching threshold, the system will not blindly execute expression switching, but will instead initiate the following closed-loop control: Feedback collection: The system extracts the difference features between the current expression and the target expression in real time and generates expression deviation feedback parameters.

[0065] Weight redistribution: Feedback parameters are applied to continuous modeling units to suppress the dominant role of a single sentiment dimension and redistribute the semantic contribution weights of each dimension in the current context.

[0066] Proportion Correction: Based on the adjusted weights, the response proportions of emotional dimensions such as "joy, surprise, and contemplation" in the target expression are readjusted to generate more stable transition control parameters and prevent the robot from making exaggerated expressions that do not fit the current atmosphere.

[0067] Microscopic control over the convergence of abrupt changes in facial expressions between frames; To ensure the robot's visual comfort during high-speed voice interaction, the convergence control of short-term facial expression changes in step S300 is specifically implemented through the following logic: The system acquires facial muscle pose data in consecutive frames at a fixed sampling period (e.g., 60 frames). When the system detects that the amplitude of a movement in a certain frame (e.g., the speed at which eyebrows are raised) exceeds the physiological limits of the human face or the high-frequency jitter threshold, the system automatically locks that frame and several adjacent frames before and after it as the "abrupt change interval".

[0068] Centered on the current frame, the system applies a weighted temporal attenuation mechanism to gradually disperse the energy of abrupt peaks to adjacent frames on both sides, reducing the energy of high-frequency jumps, and finally outputting a continuous motion effect driving signal that conforms to both the semantic expression rhythm and the biological muscle contraction law.

[0069] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A semantically driven real-time facial expression generation method for robots, characterized in that, Includes the following steps: Acquire user voice interaction data; Extract contextual semantic information from the user's voice interaction data; continuously model the robot's facial expression change process based on the contextual semantic information to obtain the semantic contribution weights of different emotional dimensions; Based on the semantic contribution weight, the response transition state in the robot's facial expression transition is constrained, and the short-term facial expression change of the robot is converged by combining the constrained response transition state, and the corresponding target facial expression driving signal is output. Based on the deviation between the current robot expression and the expected robot expression, an expression adaptation score is obtained by comparing the target expression driving signal. This expression adaptation score is then used to determine the degree of matching between the current robot expression and the user's semantics. When the matching degree reaches the preset expression switching threshold, the current robot expression is used as the transition benchmark expression for smooth expression migration, and the target expression driving signal is combined with semantic emotion inertia compensation to update the corresponding robot facial expression in real time.

2. The method according to claim 1, characterized in that, The specific process for capturing contextual semantic information is as follows: The user's voice interaction data is converted into a corresponding voice-text sequence; Based on the semantic interaction relationship in the current dialogue scenario, the semantic representation vector is obtained by matching the speech text sequence, and the current user interaction intent is determined by combining the semantic representation vector to generate contextual semantic information.

3. The method according to claim 1, characterized in that, The specific process of continuously modeling the changes in the robot's facial expressions is as follows: Extract the initial facial expression state parameters corresponding to different emotional dimensions; The initial facial expression state parameters are mapped to a preset emotional dimension space, the current displayed facial expression state is associated with the historical displayed facial expression state, and the dynamic influence weight of different semantic emotional dimensions on the change process of facial expression display parameters is determined based on the response state during the facial expression display interaction process, so as to obtain the corresponding semantic contribution weight.

4. The method according to claim 1, characterized in that, The specific process of constraining the robot's facial expression transition state by the semantic contribution weight is as follows: The facial expression display parameters corresponding to different emotional dimensions are dynamically weighted according to the semantic contribution weight to determine the degree of influence of each emotional dimension on the facial expression transition process, and the transition state adjustment parameters are generated according to the degree of influence. Based on the transition state adjustment parameters, the display frame change differences and expression migration speed in the response transition state are adjusted, and the expression migration trajectory is corrected according to the semantics of the expression display state to constrain non-semantic related expression mutation behavior.

5. The method according to claim 1, characterized in that, The specific process of comparing the deviation between the current robot expression and the expected robot expression is as follows: When the deviation between the robot's expression and the expected robot expression meets a preset condition, the expression matching confidence score between the current robot expression and the expected robot expression is obtained, and an expression adaptation score is generated based on the expression matching confidence score; when the deviation between the robot's expression and the expected robot expression does not meet the preset condition, the image deviation between the currently displayed expression frame and the target displayed expression frame is obtained, and the robot expression display status is updated based on the deviation.

6. The method according to claim 1, characterized in that, The specific process of combining semantic emotion inertial compensation with the target facial expression driving signal is as follows: Obtain the historical semantic sentiment state sequence and the current semantic sentiment state parameters; Based on the historical semantic emotion state sequence, determine the semantic inertia influence parameters in the facial expression response process; Based on the semantic emotion state parameters, the emotional response intensity corresponding to the current interaction semantics is mapped to generate target expression-driven correction parameters; The continuous facial expression display state is reconstructed by combining the semantic inertia influence parameters and the target expression driving correction parameters, and the corresponding robot facial expressions are updated in real time.

7. The method according to claim 1, characterized in that, The specific process of the convergence robot's short-term facial expression change is as follows: Obtain the facial expression response status between adjacent frames; Based on the expression response state, the abrupt change interval in the expression response state is determined. The expression jump amplitude between adjacent display frames within the abrupt change interval is converged with the current display frame as the center. The converged smooth sequence is then mapped to continuous expression display parameters, and the target expression driving signal is output.

8. The method according to claim 1, characterized in that, When the matching degree does not reach the preset expression switching threshold: Obtain facial expression deviation feedback parameters; Based on the expression deviation feedback parameters, the semantic contribution weights are reallocated, and the emotional dimension response ratio corresponding to the target expression state is corrected based on the allocated semantic contribution weights to generate robot expression transition control parameters.

9. The method according to claim 1, characterized in that, The specific process of using the current robot expression as a transitional reference expression for smooth expression transfer is as follows: Based on the target expression driving signal, the target facial motion parameters are determined, and a continuous migration trajectory between the initial state and the target state is established. The initial state of the facial expression migration is confirmed based on the continuous migration trajectory, and the continuity of the inter-frame transition in the initial state of the facial expression migration is constrained. Smooth migration control parameters are generated to execute the continuous transition from the current facial expression state to the target facial expression state.

10. A semantically driven real-time facial expression generation system for robots, used to perform the method as described in any one of claims 1-9, characterized in that, include: The voice acquisition unit is used to acquire user voice interaction data; The semantic parsing unit is used to extract contextual semantic information from the user's voice interaction data; Based on the contextual semantic information, the process of facial expression change of the robot is continuously modeled to obtain the semantic contribution weights of different emotional dimensions. The expression-driven signal generation unit constrains the response transition state in the robot's expression transition based on the semantic contribution weight, and converges the robot's short-term expression change according to the constrained response transition state, and outputs the corresponding target expression-driven signal. The expression matching and verification unit compares the deviation between the current robot expression and the expected robot expression based on the target expression driving signal to obtain an expression adaptation score, and combines the expression adaptation score to determine the degree of matching between the current robot expression and the user's semantics. When the matching degree reaches the preset expression switching threshold, the current robot expression is used as the transition benchmark expression for smooth expression migration, and the target expression driving signal is combined with semantic emotion inertia compensation to update the corresponding robot facial expression in real time.

Images