A digital human interaction wake-up and personalized response method and system based on visual perception

Abstract

The application discloses a kind of digital human interaction wake-up and personalized response method and system based on visual perception, it is related to electric digital data processing and man-machine interaction technical field.The method includes: obtaining visual behavior information in continuous image stream, determining user visual attention state and interaction candidate state based on the visual behavior information;Obtain the time correlation of the interaction candidate state Multimodal perception information, and the multimodal perception information is time series alignment and semantic recognition;Determine the interpretable high-level intent symbol;Based on the high-level intent symbol and historical symbol sequence, construct current interaction context According to the structured driving information control target response content The synchronous output of voice, expression, mouth shape and action, and update symbol history and personalized archives based on interaction result.The application can realize instructionless visual wake-up, interpretable multimodal intent understanding, low-latency personalized response and safe and controllable closed-loop learning.

Description

Technical Field

[0001] This invention relates to the fields of electronic digital data processing and human-computer interaction technology, and in particular to a method and system for digital human interaction wake-up and personalized response based on visual perception. Background Technology

[0002] With the development of artificial intelligence, digital humans have evolved into intelligent interactive agents capable of participating in tasks such as business processing and information consultation. Existing systems typically rely on wake words, key clicks, or other methods to initiate interaction, and then generate responses through speech recognition and large language models. However, existing technologies have the following shortcomings:

[0003] First, the interactive wake-up is unnatural and fails to fully utilize visual attention, resulting in a stiff wake-up and a high false wake-up rate. Second, the multimodal understanding depth is insufficient, relying only on feature-level concatenation, making it difficult to handle deep intent reasoning when verbal expression conflicts with visual behavior. Third, personalized responses lack security and controllability; the large language model suffers from issues such as factual illusions, unauthorized responses, and leakage of sensitive content, and personalization is based solely on template replacement. Fourth, the edge-cloud architecture struggles to balance real-time performance, accuracy, and privacy protection. Fifth, the system primarily responds passively, lacking the ability to proactively clarify and guide.

[0004] Therefore, there is an urgent need for a technical solution that uses visual perception as a natural trigger entry point, neural symbol fusion to achieve interpretable intent understanding, retrieval-enhanced generation and security guardrails to achieve secure and controllable response, and cloud-edge-device collaboration to achieve low latency and privacy protection. Summary of the Invention

[0005] This invention provides a method and system for wake-up and personalized response of digital human interaction based on visual perception, in order to solve the problems of unnatural wake-up methods, unexplainable multimodal understanding, insufficient security of personalized response, difficulty in balancing real-time performance and accuracy in edge-cloud architecture, and lack of continuous evolution capability in existing digital human interaction systems.

[0006] In a first aspect, embodiments of the present invention provide a method for digital human interaction wake-up and personalized response based on visual perception, including:

[0007] Acquire visual behavior information from a continuous image stream, and determine the user's visual attention state and interaction candidate state based on the visual behavior information;

[0008] When the interaction candidate state meets the wake-up condition, multimodal perception information associated with the interaction candidate state time is acquired, and the multimodal perception information is time-aligned and semantically recognized.

[0009] The continuous features obtained from semantic recognition are converted into discrete symbol tuples, and rule reasoning and confidence fusion are performed on the symbol tuples to determine interpretable high-level intent symbols.

[0010] The current interaction context is constructed based on the high-level intent symbols and historical symbol sequences, and symbol-guided retrieval is performed according to the current interaction context to obtain candidate fact information;

[0011] Personalized prompts are constructed based on the high-level intent symbols, the current interaction context, the candidate fact information, and the user's personalized profile. Candidate responses are generated and security reviews are performed to determine the target response content and structured driving information.

[0012] Based on the structured driving information, the synchronous output of voice, facial expressions, lip movements and actions corresponding to the target response content is controlled, and the symbol history and personalized profile are updated based on the interaction results.

[0013] Optionally, the step of acquiring visual behavior information from a continuous image stream and determining the user's visual attention state and interaction candidate state based on the visual behavior information includes:

[0014] Extract at least one visual behavioral feature from a continuous image stream, including facial feature points, gaze direction, head posture, mouth movement state, and hand gestures.

[0015] The visual behavior features are normalized to obtain visual state parameters corresponding to gaze duration, gaze offset angle, head orientation stability, mouth movement amplitude, and gesture semantic confidence.

[0016] Calculate the visual attention score based on the aforementioned visual state parameters;

[0017] When the visual attention score meets the preset attention conditions, and at least one of the mouth activity state, gesture action or head posture meets the interaction triggering conditions, the user is determined to be in an interaction candidate state.

[0018] Optionally, the step of acquiring multimodal perception information associated with the interaction candidate state time, and performing temporal alignment and semantic recognition on the multimodal perception information, includes:

[0019] The forward and backward time windows are determined based on the triggering time of the interactive candidate states.

[0020] At least two types of data from image segments, audio segments, and interaction logs within the forward and backward time windows are acquired as the multimodal perception information.

[0021] The multimodal sensing information is subjected to timestamp correction, sampling rate unification, noise suppression, and missing segment completion.

[0022] Visual semantic recognition, speech-text recognition, emotion polarity recognition, and behavioral intent recognition are performed on the processed multimodal perception information to obtain semantic recognition results corresponding to different modalities.

[0023] Optionally, the step of converting the continuous features obtained from semantic recognition into discretized symbol tuples, and performing rule reasoning and confidence fusion on the symbol tuples to determine interpretable high-level intent symbols includes:

[0024] The visual semantic recognition results are converted into visual symbol tuples that include entities, actions, postures, expressions, sentiment labels, confidence scores, and timestamps.

[0025] The speech-to-text recognition results are converted into text symbol tuples that include entities, speech acts, text content, sentiment polarity, keywords, confidence scores, and timestamps.

[0026] Align the visual symbol tuples and the text symbol tuples according to temporal overlap, semantic consistency, and confidence.

[0027] Logical reasoning is performed on the aligned symbol tuples based on preset reasoning rules, historical interaction rules and scenario constraint rules to obtain high-level intent symbols;

[0028] The high-level intent symbols include at least one of intent type, intent object, emotional intensity, interaction urgency, confidence level, and inference evidence chain.

[0029] Optionally, logical reasoning is performed on the aligned symbol tuples based on preset reasoning rules, historical interaction rules, and scenario constraint rules to obtain high-level intent symbols, including:

[0030] When visual symbol tuples and text symbol tuples meet the same sentiment condition within a preset time window, the confidence of the candidate intent corresponding to the same sentiment condition is increased.

[0031] When there is a semantic conflict between visual symbol tuples and text symbol tuples, the dominant modality is determined based on modal reliability, scene sensitivity, and user's historical expression habits, and the confidence of conflicting candidate intents is reduced.

[0032] When the confidence level of a candidate intent is lower than the clarification threshold, a high-level intent symbol is generated for proactive clarification.

[0033] When the confidence level of a candidate intent is not lower than the confirmation threshold, output the high-level intent symbol and its inference evidence chain corresponding to the candidate intent with the highest confidence level.

[0034] Optionally, the step of constructing the current interaction context based on the high-level intent symbols and historical symbol sequences, and performing symbol-guided retrieval according to the current interaction context to obtain candidate fact information, includes:

[0035] The short-term symbolization context is obtained by selecting symbol records from the historical symbol sequence that meet the relevant conditions in terms of time distance, intent type, and intent object with the high-level intent symbol.

[0036] The high-level intent symbol, the short-term symbolic context, and the scene identifier are combined into symbolic query conditions;

[0037] A retrieval vector is generated based on the symbolic query conditions, and the retrieval results are reordered according to intent type, intent object, sentiment intensity, and scene identifier.

[0038] From the reordered search results, select content fragments that meet the factual credibility and permission conditions as candidate factual information.

[0039] Optionally, the step of constructing personalized prompts based on the high-level intent symbols, the current interaction context, the candidate factual information, and the user's personalized profile, generating candidate responses and performing security reviews, and determining the target response content and structured driving information includes:

[0040] Obtain long-term preference summaries, interaction style tags, sensitive content constraints, and historical satisfaction records corresponding to the user identifier, and use them as the user's personalized profile;

[0041] The personalized generated prompts are constructed according to the system instruction area, behavior boundary area, user personalization area, short-term context area, retrieval fact area, and current intent area;

[0042] Based on the personalized prompts, candidate responses are generated;

[0043] The candidate responses are subjected to input compliance verification, factual consistency verification, sensitive content verification, domain compliance verification, and output usability verification.

[0044] When the candidate response passes the security review, the candidate response is determined as the target response content, and structured driving information matching the target response content is generated.

[0045] Secondly, embodiments of the present invention provide a visually perceptive-based digital human interaction wake-up and personalized response system, the system being used to execute the visually perceptive-based digital human interaction wake-up and personalized response method described in any embodiment of the present invention, the system comprising:

[0046] The visual perception module is used to acquire visual behavior information in a continuous image stream and determine the user's visual attention state and interaction candidate state based on the visual behavior information.

[0047] A multimodal recognition module is used to acquire multimodal perception information associated with the time of the interaction candidate state when the interaction candidate state meets the wake-up condition, and to perform temporal alignment and semantic recognition on the multimodal perception information.

[0048] The neural symbol fusion module is used to convert continuous features obtained from semantic recognition into discrete symbol tuples, perform rule reasoning and confidence fusion on the symbol tuples, and determine interpretable high-level intent symbols.

[0049] The context retrieval module is used to construct the current interaction context based on the high-level intent symbols and historical symbol sequences, and to perform symbol-guided retrieval according to the current interaction context to obtain candidate fact information;

[0050] The personalization generation and security review module is used to construct personalized generation prompts based on the high-level intent symbols, the current interaction context, the candidate fact information and the user's personal profile, generate candidate responses and perform security reviews, and determine the target response content and structured driving information;

[0051] The synchronous presentation module is used to control the synchronous output of voice, facial expressions, lip movements and actions corresponding to the target response content based on the structured driving information;

[0052] The closed-loop learning module is used to update symbol history and personalized profiles based on interaction results.

[0053] Thirdly, embodiments of the present invention provide an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the visual perception-based digital human interaction wake-up and personalized response method according to any embodiment of the present invention.

[0054] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute and implement the visual perception-based digital human interaction wake-up and personalized response method described in any embodiment of the present invention.

[0055] The technical solutions of the embodiments of the present invention have at least the following beneficial effects:

[0056] (1) By acquiring visual behavior information from a continuous image stream and making wake-up judgments based on visual attention state and interactive candidate state, the digital human can proactively enter a potential interactive state when natural behaviors such as continuous staring, turning head, opening mouth, nodding, and waving occur, reducing the user's dependence on fixed wake words and click entry points, and improving the naturalness and accuracy of interaction initiation. By performing temporal alignment and semantic recognition on multimodal perception information such as vision, speech, text, and interaction logs, and then converting continuous features into discrete symbol tuples, and generating high-level intent symbols based on rule reasoning and confidence fusion, this invention can transform uninterpretable neural network features into auditable, traceable, and combinatorial reasoning symbolic expressions, thereby improving the accuracy and interpretability of intent understanding in complex scenarios.

[0057] (2) By constructing the current interaction context based on high-level intent symbols and performing symbol-guided retrieval, the enhanced retrieval generation process not only focuses on literal keywords but also on the user's deep intent, emotional intensity, and interaction scenario. This makes the candidate factual information more aligned with the user's real needs and reduces the probability of off-topic or unreliable responses. By constructing personalized generation prompts from system instructions, behavioral boundaries, long-term personalized profiles, short-term symbolic contexts, retrieval facts, and current high-level intents, and by conducting multi-layered security reviews of candidate responses, this invention can achieve empathic and personalized responses by utilizing the generation capabilities of a large language model, and can also reduce the risks of illusion, unauthorized, and non-compliant outputs through factual consistency verification, sensitive content verification, and domain compliance verification.

[0058] (3) By uniformly controlling the synchronous output of speech, facial expressions, lip movements, and actions based on structured driving information, this invention enables the digital human's speech tone, facial expressions, lip movements, and body movements to be presented in a coordinated manner under the same response intent, thereby improving the consistency, realism, and user acceptance of the digital human's output. By updating symbolic history and personalized profiles based on interaction results, and combining federated aggregation and rule evolution mechanisms, this invention can continuously optimize personalized models and symbolic reasoning rules without directly aggregating original sensitive data, thereby enhancing the system's long-term adaptability in different business scenarios.

[0059] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 A flowchart of a visual perception-based digital human interaction wake-up and personalized response method provided in Embodiment 1 of the present invention;

[0062] Figure 2 This is a framework diagram of a visual perception-based digital human interaction wake-up and personalized response system provided in an embodiment of the present invention. Detailed Implementation

[0063] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0064] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0065] Application Overview:

[0066] Existing digital human systems primarily employ a linear process of "voice or click wake-up, speech recognition, text understanding, retrieval or generation, and speech and animation output." While this process is effective for explicit input, it lacks sufficient technical support for the numerous nonverbal signals, ambiguous expressions, modal conflicts, user emotional changes, and complex task collaborations present in face-to-face interactions. This invention uses visual perception as the core trigger and understanding medium for digital human interaction, employing user gaze, head orientation, mouth movements, gestures, and facial expressions as crucial evidence of potential interactions. Furthermore, it utilizes neural symbol fusion to map continuous features identified by the neural network into inferable symbolic tuples. Then, through rule-based reasoning, temporal alignment, and confidence fusion, it outputs high-level intent symbols. Finally, it uses these high-level intent symbols to drive enhanced retrieval generation, security review, and multimodal presentation of the digital human.

[0067] The technical concept of this invention is not simply to chain visual recognition, speech recognition, large language models, and digital human rendering together, but rather to enable synergy among these processes. Specifically, visual perception results are used not only for wake-up but also to determine emotion, engagement, and the intensity of proactive guidance; symbolic reasoning results are used not only for intent classification but also to guide retrieval reordering, prompt construction, and security review; retrieval facts are used not only to generate answers but also for factual consistency verification; and interaction results are used not only for archiving but also for federated learning, long-term memory updates, and the evolution of reasoning rules. Through this synergistic mechanism, the system can achieve a complete technical closed loop of "natural wake-up, interpretable and understandable, secure generation, synchronous presentation, and closed-loop evolution."

[0068] In its engineering implementation, this invention adopts a three-tiered collaborative approach involving cloud, edge, and endpoint. The user side is suitable for performing high-frequency, lightweight, and privacy-sensitive visual attention assessments; the edge side is suitable for performing low-latency, multimodal alignment, symbolic fusion, and digital human temporal control; and the cloud side is suitable for performing complex retrieval, knowledge updates, large model generation, security auditing, and federated aggregation. This division of labor follows the principles of proximity processing, on-demand uploading, minimization of sensitive data, and centralized computation of complex cognition, achieving a better balance between response latency, understanding accuracy, computational cost, and privacy protection.

[0069] Example 1:

[0070] Figure 1 This is a flowchart illustrating a visual perception-based digital human interaction wake-up and personalized response method according to Embodiment 1 of the present invention. This embodiment is applicable to scenarios requiring natural interaction by digital humans, such as government services, financial services, medical guidance, exhibition hall explanations, educational companionship, intelligent customer service, home terminals, and in-vehicle cockpits. Figure 1 As shown, the method includes:

[0071] S110. Obtain visual behavior information from a continuous image stream, and determine the user's visual attention state and interaction candidate state based on the visual behavior information.

[0072] S120. When the interaction candidate state meets the wake-up condition, acquire the multimodal perception information associated with the interaction candidate state time, and perform temporal alignment and semantic recognition on the multimodal perception information.

[0073] S130. The continuous features obtained from semantic recognition are converted into discrete symbol tuples, and rule reasoning and confidence fusion are performed on the symbol tuples to determine interpretable high-level intent symbols.

[0074] S140. Construct the current interaction context based on the high-level intent symbols and historical symbol sequences, and perform symbol-guided retrieval according to the current interaction context to obtain candidate fact information.

[0075] S150. Based on the high-level intent symbol, the current interaction context, the candidate fact information, and the user's personalized profile, a personalized generation prompt is constructed, a candidate response is generated and a security review is performed, and the target response content and structured driving information are determined.

[0076] S160. Based on the structured driving information, control the synchronous output of the voice, facial expression, lip movement and action corresponding to the target response content, and update the symbol history and personalized profile based on the interaction results.

[0077] In this embodiment, visual behavioral information can be understood as observable behavioral information expressed by the user before and after interacting with the digital human through facial, eye, head, mouth, hand, and body posture. This visual behavioral information is not directly equivalent to identity recognition information; its focus is on determining whether the user is paying attention to the digital human, whether they are likely to initiate interaction, and their emotional and engagement state. Through lightweight analysis of continuous image streams, this embodiment can identify potential interaction signals even before the user explicitly utters a wake word or clicks a button.

[0078] Example 2:

[0079] The technical solution in this embodiment is a further refinement based on the above embodiments.

[0080] In step S110, acquiring visual behavior information from a continuous image stream and determining the user's visual attention state and interaction candidate state based on the visual behavior information includes:

[0081] Extract at least one visual behavioral feature from a continuous image stream, including facial feature points, gaze direction, head posture, mouth movement state, and hand gestures.

[0082] The continuous image stream, a sequence of images continuously captured by a camera at a preset frame rate (e.g., 5-30 frames per second), is the raw data source for visual behavior information. Facial feature points refer to the coordinates of key facial features, such as the corners of the eyes, mouth, and nose. These are used to locate and track subtle facial movements. Gaze direction refers to the orientation of the user's eyes or pupils, used to determine where the user is looking, for example, whether they are looking towards the area where the digital human is located. Head posture refers to the three-dimensional rotation angle of the user's head, typically including yaw angle (turning left or right), pitch angle (looking up or down), and roll angle (tilting to the side); reflecting the user's overall attention orientation. Mouth activity refers to the opening and closing and movement of the user's lips, an important indicator of whether the user is speaking or preparing to speak. Gestures refer to semantically specific hand gestures, such as waving, raising, or pointing. Visual behavior features refer to quantitative or categorical data about user behavior extracted directly from the image stream or preliminarily calculated; this is a collective term for the aforementioned facial feature points, gaze direction, etc. Visual state parameters refer to standardized values ​​obtained by processing and normalizing the original "visual behavioral features," used to quantify a specific behavioral state. They are the basis for calculating attention scores.

[0083] Specifically, visual processing units such as MediaPipe, lightweight convolutional neural networks (CNNs), or dedicated chips are used. These models are optimized to run in real time on resource-constrained devices such as mobile phones and edge computing boxes.

[0084] Images are extracted frame-by-frame or in mini-batch from a continuous image stream. A lightweight model processes each frame in parallel or sequentially to locate the face and hand regions. Dozens of facial feature points are extracted from the face region; key hand joints are extracted from the hand region. The gaze vector is estimated based on key eye points, such as the relative positions of the pupil and corners of the eyes. The three-dimensional rotation angles of the head (yaw, pitch, roll) are calculated based on the correspondence between the 3D face model and feature points in the 2D image. The distance or rate of change between key points of the upper and lower lips is calculated to determine whether the lip is closed, slightly open, or fully open. Based on the spatial configuration of the hand key points, a classifier determines the gesture category, such as no gesture, waving, or OK gesture; and outputs the gesture semantic confidence score.

[0085] For each frame or a short segment, the model outputs a set of visual behavioral feature data containing timestamps, such as: {timestamp T, facial feature points [array], gaze vector [x,y,z], head pose angle [yaw, pitch, roll], mouth opening: 0.8, gesture: waving (confidence 0.92)}.

[0086] The visual behavior features are normalized to obtain visual state parameters corresponding to gaze duration, gaze offset angle, head orientation stability, mouth movement amplitude, and gesture semantic confidence.

[0087] Among them, visual state parameters refer to standardized values ​​obtained after processing and normalizing the original visual behavioral features, used to quantify specific behavioral states, and are the basis for calculating attention scores. Fixation duration refers to the length of time the user's gaze remains continuously on the digital human interaction area. Gaze offset angle refers to the angle between the user's gaze direction and the center direction of the digital human interaction area. Head orientation stability refers to the degree to which the user's head orientation remains stable over a period of time. Mouth movement amplitude refers to the degree of change in the opening and closing or movement of the user's lips. Gesture semantic confidence refers to the confidence score of the gesture action recognized by the system as belonging to a specific interaction intention, such as "waving".

[0088] Specifically, it's the process of transforming raw features into comparable, computable standardized parameters. Fixation duration ( Calculate and count the duration for which the user's gaze continuously falls on the digital human area. Normalization That is, a gaze span of less than 0.3 seconds is scored as 0 points, and a gaze span of more than 3 seconds is scored as 1 point, with a linear proportional mapping from 0.3 to 3 seconds to 0 to 1. 0.3 seconds and 3 seconds are the upper and lower limits of the valid range defined in the document.

[0089] Normalized value of line of sight offset angle ( Calculation: Calculate the angle between the actual line of sight and the center line of the digital human. Normalization: included angle At 0 degrees, perfectly aligned. The value is 1; when the included angle is greater than or equal to 30 degrees, It is 0; it decreases linearly between 0 and 30 degrees. 30 degrees is the defined maximum effective offset angle.

[0090] Head orientation stability ( Calculate the variances of the head's yaw, pitch, and roll angles within a sliding time window. Normalization: . This is the preset normalized reference variance. The smaller the variance (the more stable the head). The closer it is to 1.

[0091] Mouth movement range Calculate the change in distance between key points on the upper and lower lips within a short time window. Normalization: . This is a reference range, which can be set through initial calibration or empirical values. The greater the change, The higher the value, the more likely it is that someone is speaking.

[0092] Gesture semantic confidence ( The confidence score for interactive gestures, such as waving, directly using the output of the gesture recognition model is usually between 0 and 1.

[0093] Calculate the visual attention score based on the aforementioned visual state parameters;

[0094] Among them, the visual attention score refers to a single score that integrates multiple visual state parameters, used to quantify the user's current overall visual attention level towards the digital human.

[0095] Specifically, the fixation duration, gaze deviation angle, head orientation stability, mouth movement amplitude, and gesture semantic confidence are dimensionless; the dimensionless parameters are then weighted and summed according to preset weights to obtain the visual attention score. ). formula:

[0096] =w1* +w2* +w3* +w4* +w5* ;

[0097] in, , , , , These are the five normalized visual state parameters obtained in the previous step. The effective range for the gaze duration is 0.3 to 3 seconds, the effective range for the gaze deviation angle is 0 to 30 degrees, the effective range for head orientation stability is 0 to 1, the effective range for mouth movement amplitude is 0 to 1, and the effective range for gesture semantic confidence is 0 to 1. When the visual attention score is not less than a preset score threshold and continues to reach a preset time threshold, the preset attention condition is determined to be met. w1, w2, w3, w4, and w5 are preset weight coefficients, and w1 + w2 + w3 + w4 + w5 = 1. The weights reflect the importance of different behaviors in judging attention. Example of weighting settings: gaze duration (w1): 0.25–0.40 (highest weight, because sustained gaze is a strong attention signal); gaze deviation angle (w2): 0.20–0.35; head orientation stability (w3): ​​0.10–0.25; mouth movement amplitude (w4): 0.05–0.20; gesture semantic confidence (w5): 0.05–0.20.

[0098] Results: Va is a value between 0 and 1. The higher the value, the more focused the user's visual attention is and the more likely they are to want to interact with the digital human.

[0099] When the visual attention score meets the preset attention conditions, and at least one of the mouth activity state, gesture action or head posture meets the interaction triggering conditions, the user is determined to be in an interaction candidate state.

[0100] The preset attention conditions refer to the threshold conditions pre-set by the system to determine whether the visual attention score has reached a noteworthy level. These typically include a score threshold and a duration threshold; for example, a score > 0.6 lasting more than 1 second. Interaction trigger conditions refer to the judgment conditions set for specific interactive behaviors, such as mouth movements, specific gestures, nodding, etc. For example, detecting a "mouth opening" action or a "waving" gesture with a confidence level greater than 0.7. Interaction candidate states refer to the intermediate state where the system determines the user may want to initiate an interaction, but has not yet fully confirmed it. In this state, the system begins to prepare and collect more modal information, such as voice, to prepare for a possible formal interaction, but does not immediately provide a complete response. This is a key design feature aimed at balancing natural wake-up and preventing false triggers.

[0101] Specifically, it determines whether the preset attention conditions are met, and scores visual attention (...). The score should be no less than a rating threshold, for example, A1 = 0.65, and this state should last for a time threshold, for example, L1 = 1.2 seconds. Ensure that the user's attention is continuous and focused, not just a fleeting glance. This will transition the system from an idle state to a visually engaged state.

[0102] To determine if the interaction trigger condition is met, while maintaining visual attention, at least one explicit intention to interact is detected: mouth movement, and the amplitude of mouth movement (…). If the value exceeds a threshold, such as 0.35, and persists for a certain period of time, it indicates that the user may want to speak. Gestures are also considered; specific gestures, such as waving, are recognized, and their semantic confidence (...) If the value exceeds a threshold, such as 0.75, the head posture is detected as a periodic change in head pitch angle that matches a "nodding" pattern.

[0103] When both conditions are met simultaneously, the system determines that the user is in an interaction candidate state. The system state formally transitions from the visual attention state to the interaction candidate state. Upon entering this state, the system will not respond immediately, but will initiate the collection and analysis of multimodal information (such as speech) within the relevant time period to prepare for a possible formal dialogue. If a clear speech initiation or other confirmation signal is subsequently detected, the system will transition to the formal wake-up state.

[0104] In this embodiment, the contradiction between the naturalness of visual wake-up and the control of false wake-ups is resolved by using visual attention scoring, finite state machines, and interaction triggering conditions. Traditional solutions either require explicit user clicks or are prone to false triggers when any gaze is detected; while this invention, through joint judgment of duration, gaze angle, head stability, mouth movements, and gesture confidence, can distinguish between passing by, brief glances, viewing the screen, and genuine interaction intentions, thereby improving the naturalness, accuracy, and stability of digital human wake-up.

[0105] In step S120, when the interaction candidate state meets the wake-up condition, multimodal perception information associated with the interaction candidate state time is acquired, and the multimodal perception information is subjected to temporal alignment and semantic recognition, including:

[0106] When the interaction candidate state meets the wake-up condition, the forward time window and the backward time window are determined based on the trigger time of the interaction candidate state;

[0107] The wake-up condition refers to the conditions that further confirm the formal initiation of the interaction based on the interaction candidate state. This is usually due to a specific behavior detected in the interaction candidate state reaching a higher confidence level or lasting for a longer period of time. For example, in the interaction candidate state, a clear voice start point is detected, or the user makes a very clear waving gesture. The trigger time (Tc) refers to the precise point in time at which the system determines that the wake-up condition has been met. It serves as the time reference for subsequent multimodal data acquisition and alignment. The forward time window refers to a time window that traces back from the trigger time Tc, for example, [Tc- [, Tc], A timeframe of 0.5-3 seconds is acceptable. This is used to capture user behavior before the formal initiation of an interaction, such as gaze, mouth opening preparation, and confused expressions. The backward time window refers to a time window extending backward from the trigger time Tc, for example, [Tc, Tc + ... ], It can be 1-5 seconds. It is used to capture user expressions after the interaction is formally initiated, such as complete voice statements, subsequent gestures, or facial expressions.

[0108] Specifically, the trigger time Tc for the input interaction candidate state. Forward time window: [Tc - Backward time window: [Tc, Tc+] Parameter settings can be dynamically adjusted according to the scenario: (Forward Duration): Typically 0.5 to 3 seconds. For rapid question-and-answer scenarios: 0.8-1.5 seconds, primarily capturing the instantaneous facial expressions and mouth movements before the user speaks. For complex consultation scenarios: 1-3 seconds, to capture the user's longer period of hesitation and contemplation. (Backward duration): Typically 1 to 5 seconds. The initial value can be set to 2-3 seconds and can be dynamically extended based on the results of Voice Endpoint Detection (VAD). That is, if the user is detected still speaking, the backward window is automatically expanded until the end of the speech is detected. Two distinct time intervals are output as the basis for subsequent data capture.

[0109] At least two types of data from image segments, audio segments, and interaction logs within the forward and backward time windows are acquired as the multimodal perception information.

[0110] Multimodal perception information refers to multi-dimensional data reflecting the user's state acquired from different sensors or logs within a defined time window, and includes at least two categories. It is the raw material for intent understanding. Image clips refer to raw video frames within the time window, or data that has undergone anonymization processing, such as retaining only visual data of key points of the human skeleton. Audio clips refer to raw recordings or noise-reduced audio data within the time window. Interaction logs record user actions on the terminal, such as touching the screen, clicking buttons, page dwell, scrolling, and inputting, typically including event type and timestamps.

[0111] Specifically, after determining the time window, the corresponding data segments are extracted from different data buffers or storage. Data sources include: image / audio buffers, where front-end devices or edge nodes typically maintain a circular buffer to continuously store raw or lightly processed video and audio data from the most recent period; and interaction log streams, which are user operation event streams generated and sent in real time by business systems or front-end applications.

[0112] The system calculates [Tc- , Tc+ Within a given time range, queries are initiated to various data sources. All video and audio frames within the corresponding timestamp range are copied from the circular buffer. Event records whose timestamps fall within this window are filtered from the event stream or log database, such as: {Time: Tc-0.5s, Event: User enters the "Materials Download" page}, {Time: Tc+1s, Event: User clicks the "Help" button}. In privacy-sensitive scenarios, such as medical settings, keypoint sequences, behavioral tags, and audio features can be prioritized over uploading raw images and audio; that is, anonymized feature data is obtained instead of the original data.

[0113] The multimodal sensing information is subjected to timestamp correction, sampling rate unification, noise suppression, and missing segment completion.

[0114] Temporal alignment refers to the process of mapping multimodal data from different sources, such as cameras, microphones, and business systems, with different sampling rates and initial timestamps, onto the same high-precision time axis. This is the foundation for subsequent cross-modal correlation analysis. Timestamp correction refers to correcting time deviations caused by system clock asynchrony, acquisition start delays, etc., between different data sources, ensuring they reflect true physical time. Sampling rate unification refers to unifying data sampled at different frequencies, such as 30 frames / second for video and 16,000 samples / second for audio, to the same time step through resampling or aggregation, facilitating correlation analysis. Noise suppression refers to denoising the data, such as filtering out environmental noise in audio, smoothing jitter in visual keypoints, and filtering invalid click events in logs. Missing segment completion refers to reasonably interpolating or filling in temporary data loss during data transmission or processing to ensure the continuity of the time series.

[0115] Specifically, timestamp correction addresses the issue that the system clocks of device A's camera, device B's microphone, and server C's logging system may have millisecond-level deviations. A unified clock source is used, with all devices connected to an NTP (Network Time Protocol) server for clock synchronization. Event anchor calibration utilizes a cross-modal, precisely detectable synchronized event as an anchor. For example, if the system issues an audio-visual synchronized "wake-up prompt" at time Tc, this event will be simultaneously recorded by the camera (light), microphone (sound), and logging system. By comparing the timestamps recorded by each modality to this event, the offset between them can be calculated and compensated for in subsequent data.

[0116] A unified sampling rate addresses the issue that video at 30Hz (one frame every 33ms) and audio at 16kHz (0.0625ms per sample) cannot be directly correlated point-to-point. A unified time step is determined, for example, one analysis unit every 100ms. Resampling and aggregation are then performed: the key points of three video frames within a 100ms interval are averaged, or the middle frame is used to represent the state within that 100ms interval. The 1600 audio samples within the 100ms interval are extracted into a Mel-spectrum, or various acoustic features of the audio segment, such as energy and zero-crossing rate, are calculated. Based on the word-level timestamps output by ASR, each word is mapped to a corresponding 100ms time interval.

[0117] Noise suppression employs spectral subtraction, Wiener filtering, or deep learning denoising models to remove steady-state environmental noise and non-human voices. Kalman filtering or moving average filtering is applied to the coordinates of human keypoints in consecutive frames to smooth coordinate jitter caused by hand tremors or detection errors. Repeated click events are removed (anti-shake), and extremely short pause events caused by accidental touches are filtered out.

[0118] Missing data is filled in for short-term missing segments (<200ms). Linear interpolation is used to estimate the missing value using data from before and after the missing segment. For long-term missing segments, if the missing time exceeds a threshold, forced filling is no longer performed. Instead, the confidence level of that mode during that time period is marked as "low" or "missing," and its weight is reduced during subsequent fusion. This acknowledges the possibility of sensor occlusion or malfunction in the real world.

[0119] Visual semantic recognition, speech-text recognition, emotion polarity recognition, and behavioral intent recognition are performed on the processed multimodal perception information to obtain semantic recognition results corresponding to different modalities.

[0120] Semantic recognition refers to the preliminary and independent analysis and understanding of the raw data for each modality, extracting meaningful semantic labels or text. Visual semantic recognition refers to identifying behavioral or facial expression categories such as nodding, waving, smiling, and confusion from images. Automatic speech-to-text (ASR) refers to transcribing the speech content in audio into text. Emotional polarity recognition refers to judging the emotional tendency contained in a user's expression, such as positive, neutral, negative, anxious, and angry. Behavioral intent recognition refers to the preliminary classification of user intent based on single-modal information (mainly text and visual), such as inquiry, confirmation, refusal, and seeking help. Note that the intent here is preliminary and not the final interpretable high-level intent symbol output. Semantic recognition results refer to the outputs of the above recognition processes, usually structured data with timestamps, confidence levels, and source labels. For example: [Action: Nodding, Confidence: 0.88, Time interval: T1-T2, Source: Visual], [Text: "What should I do about this?", Confidence: 0.94, Time interval: T2-T3, Source: Speech].

[0121] Specifically, in visual semantic recognition, visual models with higher accuracy than edge-side models are deployed at the edges, such as convolutional networks (for skeletal motion), CNNs, or Transformers (for facial expressions). The input is aligned image segments or keypoint sequences. The output is a sequence of behavior / facial expression labels with timestamps and confidence levels. For example: [T1-T2: sustained gaze], [T2-T3: confused expression, 0.76], [T3-T4: nodding, 0.88].

[0122] Automatic Speech Recognition (ASR) employs an end-to-end speech recognition model, such as Conformer-Transducer. Input: Noise-reduced audio clip. Output: Transcribed text, word-level / sentence-level timestamps, and recognition confidence score. For example, text: “Where can I download this material?”, confidence score: 0.93, time interval: T2-T4.

[0123] Emotional polarity recognition employs a multi-branch fusion model. The visual branch takes facial expression and posture features as input and outputs visual emotion. The speech branch takes prosodic features such as pitch, energy, and speech rate as input and outputs speech emotion. The text branch takes ASR text as input and outputs text emotion. Fusion, through an attention fusion layer, combines the results from the three branches to output the final emotion label, such as "anxiety" and intensity. During training, the model randomly masks certain modalities to enhance robustness in the event of modality loss.

[0124] For behavioral intent recognition, the model employs a preliminary classification network. The input is a fusion of ASR text, keywords, current page identifiers, and visual action labels. The objective is to perform fast, coarse-grained intent classification, providing candidate directions for subsequent symbolic reasoning. For example, the output preliminary intent is: "Consulting materials," with a confidence level of 0.85. The behavioral intent recognition result here is not the final high-level intent, but only one of the candidate pieces of evidence for subsequent symbolic reasoning.

[0125] In this embodiment, fundamental problems such as asynchronous multimodal data, high noise levels, modality gaps, and semantic conflicts are addressed through pre- and post-time windows, timestamp correction, unified sampling rates, noise suppression, missing data completion, and multimodal semantic recognition. Traditional solutions tend to misbind visual expressions and speech content at different time points or rely solely on text while ignoring visual behavior. This embodiment unifies multimodal data onto the same timeline and outputs semantic results with source and confidence levels, enabling subsequent symbolic reasoning to clearly understand the source, timing, and confidence level of information, thereby improving the reliability of intent understanding.

[0126] In step S130, converting the continuous features obtained from semantic recognition into discrete symbol tuples, performing rule reasoning and confidence fusion on the symbol tuples to determine interpretable high-level intent symbols includes:

[0127] The visual semantic recognition results are converted into visual symbol tuples that include entities, actions, postures, expressions, sentiment labels, confidence scores, and timestamps.

[0128] Among them, a symbolic tuple is a structured data unit that discretizes the continuous output of a neural network into symbols with explicit fields that are readable by both machines and humans. It is the basic unit for logical reasoning.

[0129] Specifically, the input is a stream of visual semantic recognition results. For example: [Time: T1-T2, Behavior: Continuous gaze], [Time: T2-T3, Action: Nodding, Confidence: 0.91], [Time: T2-T3, Expression: Smiling, Confidence: 0.80]. Entity Completion: The default entity is "User". Field Mapping: The labels from the recognition results are directly filled into the corresponding fields of the symbol tuple. Action: Nodding, Expression: Smiling. Sentiment Induction: Based on the expression and action, a visual sentiment mapping table is queried to obtain the sentiment label. For example, "smiling and nodding" might be mapped to "positive". Time and Confidence Inheritance: The timestamp and confidence score from the input are used directly. Grouping and Aggregation: Multiple recognition results that occur within the same time period and describe the same entity, such as "nodding" and "smiling" within the time period T2-T3, are merged into the same visual symbol tuple.

[0130] Output structured visual symbol tuples. For example: V1 = <Entity: User, Action: null, Pose: Facing Digital Man, Expression: null, Sentiment Label: Focused, Confidence: 0.95, Timestamp: T1-T2>; V2 = <Entity: User, Action: Nodding, Pose: Facing Digital Man, Expression: Smiling, Sentiment Label: Positive, Confidence: 0.91, Timestamp: T2-T3>.

[0131] The speech-to-text recognition results are converted into text symbol tuples that include entities, speech acts, text content, sentiment polarity, keywords, confidence scores, and timestamps.

[0132] The text symbol tuple describes the symbol for a speech / text event.

[0133] Specifically, input the speech-text recognition results. For example, text: "This solution is good," sentiment polarity: positive, confidence score: 0.94, timestamp: T2-T4. Entity completion: The default entity is "user." Speech act analysis: Perform pragmatic analysis on the text to determine its function. This can be achieved through a small classification model or rules. For example, "This solution is good" is a "statement"; "What materials are needed?" is an "inquiry"; "Okay" might be a "confirmation." Text content: Directly input the ASR text. Sentiment polarity: Inherit the sentiment recognition results from the previous step. Keyword extraction: Use TF-IDF, TextRank, or simple part-of-speech tagging + filtering to extract core words such as nouns and verbs as keywords from the text. For example, extract ["materials", "downloads"] from "Where can this material be downloaded?". Confidence score: Combine ASR confidence score and sentiment classification confidence score. Timestamp: Inherit the input timestamp.

[0134] Output structured text symbol tuples. For example: T1 = <Entity: User, Speech Act: Statement, Text Content: This solution is good, Sentiment Polarity: Positive, Keywords: [Solution, Good], Confidence: 0.94, Timestamp: T2-T4>.

[0135] Align the visual symbol tuples and the text symbol tuples according to temporal overlap, semantic consistency, and confidence.

[0136] Temporal overlap refers to the degree to which two symbolic tuples, such as a visual and a textual one, overlap on the timeline; it is used to determine whether they describe events occurring at the same moment. Semantic consistency refers to the degree of semantic agreement between two symbolic tuples. For example, the action of "nodding" has high semantic consistency with the textual "yes," while "shaking one's head" has low semantic consistency with "yes." Confidence refers to the confidence score inherent in the symbolic tuple itself, as determined by the model. Alignment refers to the process of establishing temporal and semantic connections between visual and textual symbolic tuples, forming a cross-modal event pair to prepare for fusion inference.

[0137] Specifically, calculate the time overlap ( For visual symbol tuples (time interval) ) and text symbol tuples (time interval) The formula is: .if If the value is greater than a threshold (e.g., 0.2), then the two are considered to be correlated in time.

[0138] Calculate semantic consistency ( This is based on rule tables or vector similarity calculations. For example, actions like nodding and verbal acts like confirmation / statement, with high positive semantic consistency (e.g., positive polarity), show high semantic consistency (e.g., high consistency between facial expressions like confusion and verbal acts like inquiry). Facial expressions like confusion and verbal acts like confirmation and emotional polarity, with low positive semantic consistency (potential conflict), show low consistency (e.g., low consistency between facial expressions like confusion and verbal acts like confirmation and emotional polarity). A predefined inter-modal semantic consistency matrix can be used for fast lookups.

[0139] Alignment decisions, for each pair ( , ), calculate a comprehensive correlation score: α, β, and γ represent weights. Using bipartite graph maximum weight matching or a threshold method, the most relevant visual symbol tuple is found for each text symbol tuple, and vice versa, forming alignment pairs. For example, V2 and T1 are successfully aligned.

[0140] Logical reasoning is performed on the aligned symbol tuples based on preset reasoning rules, historical interaction rules and scenario constraint rules to obtain high-level intent symbols;

[0141] Among them, pre-defined inference rules refer to interpretable logical rules pre-established by domain experts. For example, IF action = nodding AND verbal act = affirmative statement THEN candidate intent = expressing agreement. Historical interaction rules refer to personalized expression habits learned from the user's past successful interactions. For example, if user X says "okay" with a hesitant expression, it usually indicates uncertainty. Scenario constraint rules refer to rules determined by the current business scenario. For example, at the confirmation node of a financial product, a low-confidence affirmative intent must trigger secondary confirmation. Logical reasoning refers to the process of deducing, inducing, or abductively reasoning from aligned symbol tuples based on the above rules to arrive at high-level conclusions. High-level intent symbols refer to the general instructions driving all subsequent modules; they are a richer and more interpretable structured expression than the initial intent. The high-level intent symbols include at least one of intent type, intent object, emotional intensity, interaction urgency, confidence level, and inference evidence chain.

[0142] Specifically, the input consists of aligned visual-text symbol pairs and the scene state. An inference engine, a lightweight inference system capable of executing if-then rules, such as Drools or a self-developed rule engine, is used. The engine matches aligned symbol tuples, such as V2 (nodding, smiling, positive) and T1 (statement, this solution is good, positive), with the scene state, such as the business node: solution confirmation, against conditions in the rule base. Candidate intents are generated; the matched rules produce candidate high-level intents and their initial confidence levels. For example, rule R1: "IF Action = nodding AND Verbal behavior = affirmative statement AND Emotion = positive THEN Candidate intent = expressing agreement, confidence level = 0.9".

[0143] The algorithm integrates modal confidence, temporal semantic consistency, and user historical habits, while subtracting modal conflict penalties. Consistent sentiment increases confidence: if visual and textual sentiments are consistent (both positive), the confidence of the corresponding intent is increased. Specifically, when visual and textual symbol tuples meet the same sentiment condition within a preset time window, the confidence of the candidate intent corresponding to that condition is increased. In case of conflict, the dominant modality is determined. If a conflict occurs, such as text saying "yes" but facial expression showing hesitation, the decision is made based on modal reliability (which modality is more accurate in the current environment), scene sensitivity (e.g., facial expressions might be more genuine in a financial setting), and user historical habits (e.g., whether the user frequently contradicts themselves), thus reducing the confidence of the conflicting intent. In other words, when visual and textual symbol tuples have semantic conflicts, the dominant modality is determined based on modal reliability, scene sensitivity, and user historical expression habits, and the confidence of conflicting candidate intents is reduced.

[0144] The decision output includes high-confidence confirmation: if the fused confidence level Pk of a candidate intent is greater than or equal to the confirmation threshold (e.g., 0.85), the high-level intent symbol is directly output along with a chain of reasoning evidence, such as which rules and symbol tuples are referenced. In other words, when the candidate intent confidence level is not lower than the confirmation threshold, the high-level intent symbol corresponding to the highest-confidence candidate intent and its chain of reasoning evidence are output. For low-confidence clarification: if the highest confidence level Pk is less than the clarification threshold (e.g., 0.65), a special high-level intent symbol is output [Intent type: Active clarification, Intent object: AAA, ...]. This drives the digital human to ask follow-up questions for confirmation, rather than blindly answering; that is, when the candidate intent confidence level is lower than the clarification threshold, a high-level intent symbol for active clarification is generated. For medium-confidence scenarios, if the confidence level is between the two, the decision is made based on the scenario risk. In high-risk scenarios, such as financial transactions, clarification is processed as needed; in low-risk scenarios, an attempt to answer is made, but uncertainty is indicated.

[0145] Output Example: Successful Reasoning: <Intent Type: Agreement, Intent Object: Option A, Sentiment Intensity: High, Interaction Urgency: Low, Confidence: 0.88, Evidence Chain: [Rule R1, Symbol V2, Symbol T1]>. Triggered Clarification: <Intent Type: Proactive Clarification, Intent Object: Confirmation, Sentiment Intensity: Hesitation, Interaction Urgency: Medium, Confidence: 0.62, Evidence Chain: [Conflict between Textual Affirmation and Visual Hesitation, High-Risk Business Node]>.

[0146] In this embodiment, the problem of interpretability of black-box output from multimodal neural networks is solved by transforming continuous features into symbolic tuples, as well as rule-based reasoning and confidence fusion. Traditional methods often only provide an intent classification result, making it difficult to explain the basis for judgment. The high-level intent symbols output by this embodiment include intent type, object, emotion, urgency, confidence level, and evidence chain, providing clear basis for subsequent retrieval, generation, security review, and manual auditing. Furthermore, in cases of modal conflict or low confidence, this embodiment does not force a deterministic response but instead uses intent clarification for risk control, improving the security and reliability of digital human interaction.

[0147] In step S140, the construction of the current interaction context based on the high-level intent symbols and historical symbol sequences, and the execution of symbol-guided retrieval based on the current interaction context to obtain candidate fact information, includes:

[0148] The short-term symbolization context is obtained by selecting symbol records from the historical symbol sequence that meet the relevant conditions in terms of time distance, intent type, and intent object with the high-level intent symbol.

[0149] The historical symbol sequence refers to the symbolic records left by the user's past interactions in the current session (or longer). These records are stored in the form of high-level intent symbols or symbolic interaction records, rather than raw audio and video, thus balancing contextual memory and privacy. The current interaction context refers to short-term background information constructed to understand the current intent, consisting of the current high-level intent symbols and relevant symbol records selected from the history. The short-term symbolic context refers to several historical records selected from the historical symbol sequence that are related to the current high-level intent in terms of time, intent type, and intent object. It provides short-term memory for the dialogue.

[0150] Specifically, input: current high-level intent symbol The historical symbol sequence list H = [h1, h2, ..., hn]. Calculate the relevance score: for each historical record hi, calculate its relevance to the current intent. Context-related rating The document provides a weighted formula:

[0151] =μ1× +μ2× +μ3× +μ4× +μ5× ;

[0152] in, (Time Distance Rating) refers to the more recent the historical event occurred, the higher the rating. For example, =1 - (Time difference / Maximum considered time window). Usually, only the most recent tens of seconds to a few minutes of history are considered. (Intent type similarity) determines whether historical intent types are related to the current intent type. For example, "consultation materials" and "consultation process" are related, while "consultation" and "complaint" are less related. It can be calculated using predefined intent graphs or vector similarity. (Intent Object Consistency) determines whether the historical intent object is the same as the current intent object or has a subordinate relationship. For example, if the historical object is "social security card" and the current object is "social security card replacement," then the consistency is high. This needs to be determined through entity links or knowledge graphs. (Scene Consistency) Determines whether the historical scene is consistent with the current scene. In continuous dialogue, the scene usually remains unchanged, so this item scores highly. (Emotional Continuity) Determines whether historical emotions are consistent with current emotions. For example, the transition from "confusion" to "anxiety" is consistent, while a sudden shift from "pleasure" to "anger" might be an inconsistent new topic. μ1 to μ5 are weighting coefficients, summing to 1. For example, in task-oriented dialogues, The weight for (object consistency) can be set relatively high.

[0153] Set a correlation threshold For example, 0.5, select all ≥ Historical records, arranged chronologically, constitute short-term symbolic contexts. There is usually a limit to the number of items selected, such as the most recent 5.

[0154] The high-level intent symbol, the short-term symbolic context, and the scene identifier are combined into symbolic query conditions;

[0155] Among them, short-term symbolic context refers to several historical records selected from the historical symbol sequence that are related to the current high-level intent in terms of time, intent type, and intent object. It provides short-term memory for the dialogue. Context identifier refers to the specific business scenario in which the current interaction occurs, such as a government service hall - social security window, a financial branch - financial consultation, or a medical triage center - internal medicine. It determines the scope of knowledge retrieved, security strategies, and expression style. Symbolic query conditions refer to a structured query object composed of high-level intent symbols, short-term symbolic context, and context identifier. It is a high-level search command for intelligent retrieval.

[0156] Specifically, enter: , , The three elements are encapsulated into a structure (such as JSON) according to a predefined structure format. This structure clearly indicates the various dimensions of the query. This symbolic query condition can be used directly for structure-based retrieval, or it can be converted into a natural language description or vector for retrieval.

[0157] A retrieval vector is generated based on the symbolic query conditions, and the retrieval results are reordered according to intent type, intent object, sentiment intensity, and scene identifier.

[0158] Here, retrieval vector refers to the numerical vector that transforms symbolic query conditions into text encoding models, used for semantic similarity searches in a vector knowledge base. Re-ranking refers to re-scoring and sorting the results list based on finer-grained conditions, such as intent matching, object matching, sentiment matching, and scene matching, after obtaining a batch of candidate knowledge fragments through initial retrieval, so as to place the results that best fit the current context at the top.

[0159] Specifically, symbolic query conditions are transcribed into a coherent natural language query. For example, in a government service hall's social security window scenario, a user wants to inquire about the materials required for reissuing a social security card. They are somewhat confused, having previously asked about the application process and the location of the social security bureau. This text is then input into a text encoder, such as BERT, to obtain the query vector. Multi-field joint encoding: Different fields in the query conditions—intent type, object, scenario, etc.—are encoded into vectors, and then passed through a fusion layer, such as concatenating them followed by a fully connected layer, to generate the final query vector.

[0160] Using the generated query vector, an approximate nearest neighbor search is performed in the vectorized knowledge base to recall the top-K (e.g., 50) semantically relevant candidate knowledge fragments, resulting in a preliminary list. Reordering: For For each candidate segment, a more refined comprehensive score is calculated. The formula is as follows:

[0161] Score = λ1 × +λ2× +λ3× +λ4× +λ5× +λ6× + λ7× +λ8× -λ9×Risk;

[0162] in, This represents the vector semantic similarity score, derived from the initial retrieval. This refers to keyword matching score, such as the matching results for keywords like "social security card" and "replacement". (Intent Type Matching) determines whether the intent type suitable for the candidate fragment matches the current intent. For example, a knowledge fragment about "materials list" is highly matched with the intent of "consultation materials". (Intent-Object Matching) determines whether the object described by the candidate fragment matches the object the user is interested in. This is crucial for resolving polysemy and ensuring that the retrieved material is for "social security card replacement" rather than "social security card application". (Scene matching degree) indicates whether the applicable scene of the candidate fragment label includes the current scene. (Emotional fit) refers to whether the expression style of the candidate segment is suitable for the current user's emotion. For example, for anxious users, content with clear steps and a reassuring tone has a higher fit. (Fact Credibility) is the source authority score of the candidate segment (preset). (Timeliness score) represents the freshness of the knowledge; the newer, the better. Risk (penalty item) deducts points if the candidate content is sensitive or incompatible with the current user's permissions. λ1 to λ9 are weight parameters that need to be optimized based on the scenario. A reordered list is obtained by calculating the score for each candidate and sorting them in descending order. .

[0163] From the reordered search results, select content fragments that meet the factual credibility and permission conditions as candidate factual information.

[0164] The factual credibility criteria refer to the standards used to assess the authority, accuracy, and timeliness of the knowledge fragments themselves. For example, content from official policy databases is more credible than content from online forums. Access control criteria refer to the security rules that determine whether a user has the right to access certain knowledge fragments based on their identity and authorization level. For example, personal social security details can only be viewed by the individual or authorized personnel. Candidate factual information refers to the knowledge content fragments that are ultimately obtained after retrieval, reordering, and filtering, and can be used to support the generation of answers. Each fragment should include metadata such as source, credibility, update time, and access control tags.

[0165] Specifically, applying factual credibility criteria, each knowledge fragment has a credibility tag when it is added to the database, such as authoritative release, expert review, user contribution, and score. The system sets a minimum credibility threshold. For example, 0.7. Only select... ≥ The system uses a knowledge fragment as an example. Permission conditions apply: each knowledge fragment also has permission tags, such as public, internal staff, and the user themselves. The system checks the permission tags based on the user's identity in the current session, such as anonymous or logged-in user A. Only fragments that the user has the right to access are selected.

[0166] From the reordered list Starting from the top, each fragment is checked sequentially to see if it passes both the credibility and permission filters. The top M (e.g., 3-5) fragments that pass the filters are selected to form the final candidate fact list, Facts. Each fragment in Facts is organized into a structured format, including fields such as content, source, credibility, update time, permission tag, and applicable scenario, for reference and tracing in subsequent generation steps.

[0167] In this embodiment, symbolic context construction and symbol-guided retrieval solve the problem that traditional retrieval relies solely on literal similarity and struggles to understand deeper intents and emotional states. Because the retrieval process is constrained by high-level intent symbols, short-term history, scene tags, and permission conditions, the system can return more accurate, compliant, and relevant candidate factual information that better reflects the user's actual needs, providing a reliable basis for subsequent personalized generation and security review.

[0168] In step S150, the step of constructing a personalized prompt based on the high-level intent symbol, the current interaction context, the candidate fact information, and the user's personal profile, generating candidate responses and performing security reviews, and determining the target response content and structured driving information includes:

[0169] Obtain long-term preference summaries, interaction style tags, sensitive content constraints, and historical satisfaction records corresponding to the user identifier, and use them as the user's personalized profile;

[0170] The user's personalized profile refers to a symbolic summary of their long-term interaction preferences and habits, created within the scope of their authorization. It is not the original dialogue record, but rather abstracted tags and statistical values ​​used to guide response generation. The long-term preference summary refers to stable preferences summarized from the user's historical interactions, such as a preference for detailed explanations or brief answers, or a preference for a formal or friendly tone. Interaction style tags describe the user's preferred interaction style, such as straightforward, cautious, or requiring multiple confirmations. Sensitive content constraints refer to the scope of content that the digital human is not allowed to mention or discuss, as set by the user or business rules. For example, a user might set not to discuss investment returns. Historical satisfaction records refer to user feedback on past digital human responses, such as statistical summaries of likes / dislikes, used to assess which types of responses are more popular with the user.

[0171] Specifically, for logged-in users, their long-term profile is retrieved from personalized cloud storage or edge cache based on their user identifier. For unlogged-in users, such as anonymous visitors, a temporary profile is created based on the current session and is deleted or anonymized after the session ends, according to established rules. Profile content structure: Profile P is a structure containing a summary of long-term preferences corresponding to the user identifier, interaction style tags, sensitive content constraints, and historical satisfaction records. Profiles are not static; updates to long-term preferences require user authorization and employ gradual methods such as exponential moving averages to avoid altering stable preferences due to a single interaction.

[0172] The personalized generated prompts are constructed according to the system instruction area, behavior boundary area, user personalization area, short-term context area, retrieval fact area, and current intent area;

[0173] The personalized generation prompt refers to a carefully structured text that integrates system instructions, safety rules, user preferences, current context, reliable facts, and deep intentions, submitted to the generative model as a task specification. This structured partitioning aims to reduce the risk of illusions and boundary violations inherent in large models. The system instruction zone defines the digital human's basic roles, responsibilities, and service scope. The behavior boundary zone explicitly lists prohibited actions and safety red lines, such as prohibiting medical diagnoses and promising specific benefits. The user personalization zone injects a summary of the current user's long-term preferences. The short-term context zone provides the symbolic dialogue history of recent rounds, giving the model short-term memory. The retrieved facts zone provides sourced candidate factual information, requiring the model to primarily rely on these facts in its responses. The current intention zone clearly elucidates the high-level intention symbols derived from reasoning, informing the model of the user's core needs and emotional state.

[0174] Specifically, through this structure, system instructions and security boundaries are strongly constrained, user personalization and current intent guide the answer style, short-term context provides coherence, and the fact retrieval area fundamentally limits the sources of generated content, transforming the “generation” behavior of the large model to a large extent into the behavior of “organizing, paraphrasing and embellishing known facts”, which is the core of the illusion of control.

[0175] Based on the personalized prompts, candidate responses are generated;

[0176] The candidate response is a preliminary answer text generated by the generative model based on personalized prompts. It has not yet passed security checks and cannot be directly output to the user.

[0177] Specifically, a large language model with secure alignment and domain fine-tuning is employed. The model deployment location can be dynamically selected based on task scheduling strategies: the cloud handles complex generation tasks requiring strong empathy or high security; the edge / local server handles simplified generation or templated responses when network conditions are poor or content is sensitive. The input is the complete, personalized generated prompt text constructed above. The model output is a structured candidate response draft (Rdraft), which includes not only text but may also contain some tags.

[0178] The candidate responses are subjected to input compliance verification, factual consistency verification, sensitive content verification, domain compliance verification, and output usability verification.

[0179] Security review: A series of automated verification processes performed on candidate responses before output, acting as multiple filters to ensure the security, reliability, and compliance of the response. Input compliance verification checks whether the user's original question contains illegal, irregular, offensive, or unsuitable content for automated answering. Fact consistency verification verifies whether the key factual statements in the candidate response are consistent with the evidence provided in the fact section, preventing the model from fabricating information (illusion). Sensitive content verification detects whether the response contains sensitive content such as personal privacy information, trade secrets, or inappropriate remarks. Domain compliance verification checks whether the response content complies with specific regulations and ethical requirements of the current business scenario, such as finance, healthcare, and government affairs. For example, investment advice cannot be given in a financial scenario. Output usability verification evaluates the fluency, tone, and length of the response text to ensure good readability and user experience.

[0180] Specifically, input compliance checks use rules or classifiers to examine the user's original questions. For example, filtering inputs containing illegal keywords, personal attacks, or requests that require the system to perform impossible tasks, such as "hacking into the system."

[0181] The factual consistency check extracts key factual statements from the answer_text, such as required materials A, B, and C, and processing time of D days; these are compared with the content in the fact retrieval area. Quantification formula. ,in It is the number of supported facts. No supporting facts were found. This is the number of conflicting facts. Set a threshold. For example, if the value is 0.9, it will not pass if it is below the threshold and needs to be rewritten.

[0182] Sensitive content verification uses a sensitive word library and named entity recognition model to check whether personal privacy information such as ID card number, mobile phone number, bank card number, and address unexpectedly appears in the response, or whether it contains content that is prohibited from being disseminated by laws and regulations.

[0183] Domain compliance verification: Different rule sets are applied based on the scenario: In finance, it checks for illegal promises such as "guaranteed principal and interest" or "guaranteed profits," and whether products were recommended without suitability assessment. In healthcare, it checks whether disease diagnoses and prescriptions were made. In government, it checks whether approvals were given without authorization. This is typically accomplished by combining rule matching and compliance classification models.

[0184] Output usability checks whether the response text is fluent and free of garbled characters, whether the tone matches the `response_emotion` setting, and whether the length is within a reasonable range, avoiding being too long or too short. `response_emotion` refers to the expected emotional state or tone label set when generating the digital human's response, used to control the emotional tendency of the output text. Common values ​​include, but are not limited to: neutral, happy, sympathetic, encouraging, and professional.

[0185] Comprehensive decision: Pass=( ≥ AND ==1) AND ( ≥ AND ≥ This means that factual consistency, access control, sensitive content security score, and domain compliance score must all meet the standards.

[0186] When the candidate response passes the security review, the candidate response is determined as the target response content, and structured driving information matching the target response content is generated.

[0187] The target response content refers to the candidate responses that have passed all security reviews and are ultimately confirmed as the answer text that can be output to the user. Structured driving information refers to the control scripts generated to achieve audio-visual synchronization for the digital human. It is not natural language text, but structured data that specifies how to perform the target response content using speech, facial expressions, lip movements, and gestures. For example, it defines which words to nod when spoken and which paragraphs to use a concerned tone.

[0188] Specifically, determine the target response content; if the candidate response passes all security reviews, then its... The field is officially identified as the target response content. If it fails, it proceeds to revision, downgrade (outputting a safer simplified version), or manual processing. Inputs include the target response content, high-level intent symbols (especially sentiment intensity), and the `drive_hint` from Rdraft. Through a series of rules and a lightweight model, a control timeline accurate to milliseconds is generated.

[0189] The speech parameters determine voice_style (friendly / serious), speech_rate, and pitch based on emotional intensity and tone preference, and predict pause_points by combining the text's grammar and stress. The lip-shape sequence converts the text into a phoneme sequence, and generates a mouth_shape_sequence that is strictly aligned with the speech timestamps based on the phoneme-lip shape mapping table.

[0190] Facial expressions and gesture sequences are segmented semantically and based on emotion, inserting facial expressions (e.g., a smile when saying "Don't worry") and gesture sequences (e.g., guiding gestures when explaining steps). The central timing controller for timeline synthesis integrates all parameters into a unified timeline, ensuring synchronized playback of speech, lip movements, facial expressions, and gestures.

[0191] This ultimately generates a structured driving information D, which directly drives the digital human rendering engine. For example: D = {timeline: [...], voice: {...}, face: {...}, gesture: {...}}.

[0192] In this embodiment, personalized prompt generation, multi-layered security review, and structured information generation solve the problems of traditional digital human responses being either rigid templates or uncontrollable large models. The system can generate natural and empathetic responses based on user preferences and current emotions, and can control the generation boundaries through candidate facts and security rules, reducing the risks of hallucinations, privacy leaks, unauthorized responses, and domain non-compliance.

[0193] In step S160, the synchronous output of voice, facial expression, lip movement and action corresponding to the target response content is controlled according to the structured driving information, and the symbol history and personalized profile are updated based on the interaction results.

[0194] Interaction results refer to the quantified and symbolic record of the outcome and effect of a complete interaction round. This includes not only whether the system successfully answered the question, but also the user's immediate feedback, such as interruptions, follow-up questions, satisfaction ratings, task completion status, and response delays. Symbolic history refers to the recent interaction sequence recorded in a structured and discretized form, such as high-level intent symbols and response summaries. As short-term memory, it provides context for the next interaction. Compared to the original log, it is more compact, more privacy-preserving, and easier for reasoning. Personalized profiles refer to a model of a user's long-term stable preferences and habits, maintained with the user's authorization, such as detailed explanations of preferences and adaptation to slower speech rates. It is progressively updated based on the results after each interaction, allowing the system to increasingly understand the user.

[0195] Specifically, the input is structured driving information (D), which includes speech parameters, lip-sync sequences, facial expression labels, action sequences, and their respective timestamps. A central timing controller, typically located at an edge computing node, parses this information to generate a master timeline with millisecond-level precision. This timeline serves as the absolute reference clock for all output modules.

[0196] The emotional speech synthesis model is invoked. The input consists of the target response content and speech parameters from D, namely emotion, speech rate, and pitch. An audio waveform with phoneme-level timestamps is generated. To achieve low latency, streaming synthesis can be used, where audio is generated while the already generated portion is being played, without waiting for the entire audio segment to be generated.

[0197] The system uses phoneme timestamps provided by speech synthesis to look up a phoneme-visual position mapping table to drive the lip-sync model of the digital human. It calculates the peak time points of lip-sync movements. ) and corresponding phoneme speech time points ( The absolute error (Eav) is calculated. The system dynamically adjusts rendering to ensure Eav is within a threshold, such as <80ms. If network or load causes latency, smooth audio and slightly delayed animations will be prioritized over the other way around.

[0198] Based on the facial expression tags and action sequences in D, the digital human's animation rendering engine is invoked. Actions can be based on pre-recorded motion capture templates or driven in real time by a lightweight motion generation model. The amplitude of the actions is normalized and controlled according to the seriousness of the scene; for example, the amplitude of actions in government affairs scenes is 0.2-0.6, and in explanatory scenes it is 0.4-0.9, to avoid exaggerated or unprofessional performances.

[0199] The timing controller strictly follows the main timeline, sending synchronization signals to the audio device and graphics rendering engine to trigger the rendering and playback of each frame. It monitors the rendering frame rate and audio buffer status in real time. If the frame rate falls below a threshold, such as 24fps, the animation complexity is automatically reduced; if the user actively interrupts, all output is immediately stopped, and a new listening state is entered.

[0200] Transform the raw facts of an interaction into a storable, auditable, and analyzable symbolic summary, avoiding the storage of raw audio and video. Append the generated symbolic interaction record L to the end of the user / session's historical symbol sequence. Set a length limit for this sequence, such as saving the most recent 20 rounds, and manage short-term memory using a rolling window to prevent unlimited data growth. Long-term preferences should not change drastically due to a single action; they should be updated gradually and smoothly. Use an exponential moving average, formula:

[0201] ;

[0202] in, The old preference values ​​are used, such as explaining the depth preference = 0.7, where 0 represents brief and 1 represents detailed. For example, if a user actively requests a more detailed explanation after this round of interaction, then... =1; if the user interrupts the lengthy explanation, then =0. ρ is the smoothing coefficient (0.7-0.95). The higher the ρ, the more conservative and stable the preference update. This is the updated preference value.

[0203] Updates are not triggered with every interaction. Updates are only triggered when the system observes multiple, consistent user behavior patterns and the user has authorized long-term memory. For example, if a user requests to speak slower after three consecutive fast-paced responses, the system will gradually lower their preferred speaking speed.

[0204] Successful samples (L) with good completion status, positive user feedback, and passed security review are marked as successful and stored in the training pool. Symbolic pattern mining is performed on these successful samples to attempt to automatically summarize new candidate inference rules. For example, if the pattern of "on the material download page, staying for more than 15 seconds, and showing a confused expression" successfully triggers the active guidance intent multiple times, a new rule candidate can be formed. After manual review and confidence verification, this candidate rule is incrementally distributed to the edge symbolic inference engine. For model parameter optimization, a federated learning paradigm can be adopted. Each edge node uses local anonymized data to calculate model updates. For example, for fine-tuning of personalized models, only encrypted model parameter updates are uploaded to the cloud for aggregation, achieving privacy-preserving and secure evolution where the data remains unchanged while the model evolves.

[0205] In this embodiment, by unifying the timeline, synthesizing emotional speech, lip-syncing, coordinating facial expressions and gestures, and updating the symbolic closed-loop, the problems of the digital human's correct but unnatural responses and the system's inability to continuously evolve from interactions are solved. By converting interaction results into symbolic records and updating short-term history and long-term archives, the system can better understand the user's continuous intent and personalized preferences in subsequent interactions, thereby forming a low-latency, highly consistent, and continuously optimized digital human interaction closed loop.

[0206] Example 3:

[0207] Figure 2 This is a framework diagram of a visually perceptual-based digital human interaction wake-up and personalized response system provided in Embodiment 3 of the present invention. Figure 2 As shown, the system includes:

[0208] The visual perception module 210 is used to acquire visual behavior information in a continuous image stream and determine the user's visual attention state and interaction candidate state based on the visual behavior information.

[0209] The multimodal recognition module 220 is used to acquire multimodal perception information associated with the time of the interaction candidate state when the interaction candidate state meets the wake-up condition, and to perform temporal alignment and semantic recognition on the multimodal perception information.

[0210] The neural symbol fusion module 230 is used to convert the continuous features obtained from semantic recognition into discrete symbol tuples, perform rule reasoning and confidence fusion on the symbol tuples, and determine interpretable high-level intent symbols.

[0211] The context retrieval module 240 is used to construct the current interaction context based on the high-level intent symbols and the historical symbol sequence, and to perform symbol-guided retrieval according to the current interaction context to obtain candidate fact information;

[0212] The review module 250 is used to construct personalized generation prompts based on the high-level intent symbols, the current interaction context, the candidate fact information and the user's personalized profile, generate candidate responses and perform security reviews, and determine the target response content and structured driving information;

[0213] The synchronous presentation module 260 is used to control the synchronous output of voice, facial expression, lip movement and action corresponding to the target response content according to the structured driving information; and to update symbol history and personalized profiles based on the interaction results.

[0214] The visual perception-based digital human interaction wake-up and personalized response system provided in the embodiments of the present invention can execute the visual perception-based digital human interaction wake-up and personalized response method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0215] Example 4:

[0216] This invention provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the Raman spectroscopy quantitative analysis method combining a large language model as described in any embodiment of this invention.

[0217] In some embodiments, a Raman spectroscopy quantitative analysis method incorporating a large language model can be implemented as a computer program. The computer program for implementing the method of the present invention can be written using any combination of one or more programming languages. In the context of the present invention, a computer-readable storage medium stores computer instructions for causing a processor to execute and implement the Raman spectroscopy quantitative analysis method incorporating a large language model provided by the present invention.

[0218] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0219] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for digital human interaction wake-up and personalized response based on visual perception, characterized in that, include: Acquire visual behavior information from a continuous image stream, and determine the user's visual attention state and interaction candidate state based on the visual behavior information; When the interaction candidate state meets the wake-up condition, multimodal perception information associated with the interaction candidate state time is acquired, and the multimodal perception information is time-aligned and semantically recognized. The continuous features obtained from semantic recognition are converted into discrete symbol tuples, and rule reasoning and confidence fusion are performed on the symbol tuples to determine interpretable high-level intent symbols. The current interaction context is constructed based on the high-level intent symbols and historical symbol sequences, and symbol-guided retrieval is performed according to the current interaction context to obtain candidate fact information; Personalized prompts are constructed based on the high-level intent symbols, the current interaction context, the candidate fact information, and the user's personalized profile. Candidate responses are generated and security reviews are performed to determine the target response content and structured driving information. Based on the structured driving information, the synchronous output of voice, facial expressions, lip movements and actions corresponding to the target response content is controlled, and the symbol history and personalized profile are updated based on the interaction results.

2. The method according to claim 1, characterized in that, The step of acquiring visual behavior information from a continuous image stream and determining the user's visual attention state and interaction candidate state based on the visual behavior information includes: Extract at least one visual behavioral feature from a continuous image stream, including facial feature points, gaze direction, head posture, mouth movement state, and hand gestures. The visual behavior features are normalized to obtain visual state parameters corresponding to gaze duration, gaze offset angle, head orientation stability, mouth movement amplitude, and gesture semantic confidence. Calculate the visual attention score based on the aforementioned visual state parameters; When the visual attention score meets the preset attention conditions, and at least one of the mouth activity state, gesture action or head posture meets the interaction triggering conditions, the user is determined to be in an interaction candidate state.

3. The method according to claim 1, characterized in that, The step of acquiring multimodal perception information associated with the interaction candidate state time, and performing temporal alignment and semantic recognition on the multimodal perception information, includes: The forward and backward time windows are determined based on the triggering time of the interactive candidate states. At least two types of data from image segments, audio segments, and interaction logs within the forward and backward time windows are acquired as the multimodal perception information. The multimodal sensing information is subjected to timestamp correction, sampling rate unification, noise suppression, and missing segment completion. Visual semantic recognition, speech-text recognition, emotion polarity recognition, and behavioral intent recognition are performed on the processed multimodal perception information to obtain semantic recognition results corresponding to different modalities.

4. The method according to claim 1, characterized in that, The process of converting continuous features obtained from semantic recognition into discrete symbol tuples, performing rule-based reasoning and confidence fusion on the symbol tuples, and determining interpretable high-level intent symbols includes: The visual semantic recognition results are converted into visual symbol tuples that include entities, actions, postures, expressions, sentiment labels, confidence scores, and timestamps. The speech-to-text recognition results are converted into text symbol tuples that include entities, speech acts, text content, sentiment polarity, keywords, confidence scores, and timestamps. Align the visual symbol tuples and the text symbol tuples according to temporal overlap, semantic consistency, and confidence. Logical reasoning is performed on the aligned symbol tuples based on preset reasoning rules, historical interaction rules and scenario constraint rules to obtain high-level intent symbols; The high-level intent symbols include at least one of intent type, intent object, emotional intensity, interaction urgency, confidence level, and inference evidence chain.

5. The method according to claim 4, characterized in that, Logical reasoning is performed on the aligned symbol tuples based on preset reasoning rules, historical interaction rules, and scenario constraint rules to obtain high-level intent symbols, including: When visual symbol tuples and text symbol tuples meet the same sentiment condition within a preset time window, the confidence of the candidate intent corresponding to the same sentiment condition is increased. When there is a semantic conflict between visual symbol tuples and text symbol tuples, the dominant modality is determined based on modal reliability, scene sensitivity, and user's historical expression habits, and the confidence of conflicting candidate intents is reduced. When the confidence level of a candidate intent is lower than the clarification threshold, a high-level intent symbol is generated for proactive clarification. When the confidence level of a candidate intent is not lower than the confirmation threshold, output the high-level intent symbol and its inference evidence chain corresponding to the candidate intent with the highest confidence level.

6. The method according to claim 1, characterized in that, The process of constructing the current interaction context based on the high-level intent symbols and historical symbol sequences, and performing symbol-guided retrieval based on the current interaction context to obtain candidate fact information includes: The short-term symbolization context is obtained by selecting symbol records from the historical symbol sequence that meet the relevant conditions in terms of time distance, intent type, and intent object with the high-level intent symbol. The high-level intent symbol, the short-term symbolic context, and the scene identifier are combined into symbolic query conditions; A retrieval vector is generated based on the symbolic query conditions, and the retrieval results are reordered according to intent type, intent object, sentiment intensity, and scene identifier. From the reordered search results, select content fragments that meet the factual credibility and permission conditions as candidate factual information.

7. The method according to claim 1, characterized in that, The process of constructing personalized prompts based on the high-level intent symbols, the current interaction context, the candidate factual information, and the user's personalized profile, generating candidate responses and performing security reviews, and determining the target response content and structured driving information includes: Obtain long-term preference summaries, interaction style tags, sensitive content constraints, and historical satisfaction records corresponding to the user identifier, and use them as the user's personalized profile; The personalized generated prompts are constructed according to the system instruction area, behavior boundary area, user personalization area, short-term context area, retrieval fact area, and current intent area; Based on the personalized prompts, candidate responses are generated; The candidate responses are subjected to input compliance verification, factual consistency verification, sensitive content verification, domain compliance verification, and output usability verification. When the candidate response passes the security review, the candidate response is determined as the target response content, and structured driving information matching the target response content is generated.

8. A visually-perceptive-based digital human interaction wake-up and personalized response system, characterized in that, The system is used to execute the visual perception-based digital human interaction wake-up and personalized response method according to any one of claims 1-7, including: The visual perception module is used to acquire visual behavior information in a continuous image stream and determine the user's visual attention state and interaction candidate state based on the visual behavior information. A multimodal recognition module is used to acquire multimodal perception information associated with the time of the interaction candidate state when the interaction candidate state meets the wake-up condition, and to perform temporal alignment and semantic recognition on the multimodal perception information. The neural symbol fusion module is used to convert continuous features obtained from semantic recognition into discrete symbol tuples, perform rule reasoning and confidence fusion on the symbol tuples, and determine interpretable high-level intent symbols. The context retrieval module is used to construct the current interaction context based on the high-level intent symbols and historical symbol sequences, and to perform symbol-guided retrieval according to the current interaction context to obtain candidate fact information; The personalization generation and security review module is used to construct personalized generation prompts based on the high-level intent symbols, the current interaction context, the candidate fact information and the user's personal profile, generate candidate responses and perform security reviews, and determine the target response content and structured driving information; The synchronous presentation module is used to control the synchronous output of voice, facial expressions, lip movements and actions corresponding to the target response content based on the structured driving information; The closed-loop learning module is used to update symbol history and personalized profiles based on interaction results.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the visual perception-based digital human interaction wake-up and personalized response method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the visual perception-based digital human interaction wake-up and personalized response method according to any one of claims 1-7.

Images