Method and system for realizing real-time two-way visual interaction of digital human by calling camera

Abstract

The invention discloses a method and system for realizing real-time two-way visual interaction of a digital human by calling a camera, and belongs to the technical field of multi-modal interaction, and the method comprises the steps: collecting video streams of a user and an environment in real time through a mobile terminal camera, and synchronously obtaining a natural language question of the user; screening related video keys based on question semantics; performing cross-modal semantic fusion analysis on the question text and the key frame image by using a multi-modal model to generate a corresponding answer result; and the virtual digital person outputs an answer through voice synthesis and synchronous anthropomorphic animation, so that real-time image-text question and answer interaction between the digital person and the user is realized. Aiming at the problem that the existing digital human interaction lacks visual situation feeling, the invention provides an end-cloud collaborative visual question-answer interaction scheme, so that the video semantic processing load of a mobile end can be effectively reduced, the end-cloud collaborative real-time image-text question-answer interaction is realized, the perception ability of the digital human to the environment is greatly improved, and the user experience is improved. And the intuition and the naturalness of user interaction experience are improved.

Description

Technical Field

[0001] The present invention relates to the field of multimodal interaction technology, and in particular to a method and system for realizing real-time two-way visual interaction of digital humans by calling a camera. Background Art

[0002] With the convergence of artificial intelligence and virtual reality technologies, digital humans, as multimodal interaction vehicles, have been widely used in scenarios such as intelligent customer service, virtual tour guides, and immersive education. Existing systems primarily achieve anthropomorphic interaction by integrating speech recognition, natural language processing, and expression-driven technologies. However, their perception capabilities are still limited to single-modal input and lack the ability to interpret environmental visual information in real time.

[0003] Current multimodal systems require users to explicitly upload static images as input, which the system then analyzes offline to generate responses. These solutions suffer from a lack of temporal continuity and a disconnected interactive process. While some research has attempted to integrate cameras into interactive systems, their functionality is limited to single-command responses. Mobile deployments face the triple challenges of processing continuous video streams exceeding the computing power of the terminal, significantly increasing network latency due to uploading raw video, and the inability to establish a semantically coordinated relationship between user questions and the real-time image.

[0004] In addition, the output of existing digital humans is mainly voice and basic facial expression animation, and lacks spatial directional feedback, making it difficult for users to establish a cognitive mapping of "what you ask is what you see".

[0005] Therefore, there is an urgent need for a low-latency, highly robust bidirectional visual interaction mechanism on mobile terminals. This mechanism, through a collaborative end-cloud architecture and semantically guided visual compression strategies, addresses the collaborative optimization of real-time performance, accuracy, and immersion. To address these challenges, the present invention proposes a method and system for enabling real-time, bidirectional visual interaction with digital humans using a camera. Summary of the Invention

[0006] Purpose of the Invention In order to solve the above problems, the purpose of the present invention is to provide a method and system for realizing real-time two-way visual interaction of digital humans by calling a camera, establishing a spatiotemporal correlation between visual flow and language interaction by realizing real-time visual analysis of the user's physical environment, and designing a semantic-guided key frame screening mechanism and a cross-modal attention fusion model to reduce the computing load of the mobile terminal and improve the accuracy of semantic understanding, thereby significantly improving the user's immersion and operational efficiency in different scenarios.

[0007] Technical Solution To achieve the above objectives, the present invention provides a method and system for realizing real-time two-way visual interaction of a digital human by calling a camera. The solution dynamically collects the environmental video stream through the terminal camera, and outputs a time-series image sequence after adaptive frame rate control and preprocessing. Based on the semantic content of the user's question, a heuristic scoring function is used to perform key frame screening on the image frames, retaining only the frame subset or its visual features that are highly relevant to the semantics of the user's question. The terminal transmits the frame subset and the user's question to the cloud, where the visual encoder extracts high-level image features and the language encoder generates text features, and achieves feature alignment and joint reasoning through a cross-modal attention mechanism to generate the answer text. After receiving the answer, the terminal drives the digital human to output the answer through speech synthesis and synchronous animation, and combines spatial feedback to complete the interaction closed loop.

[0008] In a first aspect, the present invention provides a method for implementing real-time two-way visual interaction of a digital human by calling a camera, comprising: Acquire image frames of the user and their environment in real time through the camera of the terminal device; Accepting natural language questions raised by users in voice and / or text form, and establishing semantic associations between the questions and real-time image frames; Performing visual language multimodal semantic fusion analysis based on the question and the associated image frame, and generating a corresponding answer result; The answer result is output by a virtual digital human through voice synthesis and synchronized animated expressions.

[0009] Furthermore, before establishing semantic association between the question and the real-time image frame, key frames are screened on the real-time acquisition frame sequence based on the semantic content of the question, and the key frames are highly correlated with the question semantics.

[0010] Furthermore, the multimodal semantic fusion analysis is achieved through collaboration between the terminal device and the cloud server; Among them, the terminal device performs preprocessing and feature extraction on the image frame, and transmits the extracted visual features and question text to the cloud server; the cloud server performs deep cross-modal reasoning based on the received visual features and question text to generate an answer result.

[0011] Furthermore, in the semantic fusion analysis process, the visual features of the image frame are extracted by a visual encoder, the semantic features of the question text are extracted by a language encoder, and the visual features and semantic features are fused using a cross-modal attention mechanism.

[0012] Furthermore, when the virtual digital human outputs the answer result, an anthropomorphic feedback behavior is performed synchronously; The anthropomorphic feedback behavior includes adjusting the line of sight toward the user, indicating the target object in the image frame through virtual body movements, or superimposing a target mark on the real-time image.

[0013] Furthermore, the method dynamically monitors the network connection status and image frame quality; When it is detected that the network delay exceeds the threshold or the image clarity is lower than the standard, the video acquisition parameters and inference strategy are adaptively adjusted.

[0014] In a second aspect, the present invention further provides a system for implementing real-time two-way visual interaction of digital humans by calling a camera. When the system is running, the method described in the first aspect is executed, including: A camera acquisition module, used to capture image frames of the user and their environment in real time; A question acquisition module is used to receive user questions in voice and / or text form; A key frame screening module, screening semantic key frames from real-time image frames based on the semantic content used to ask questions; A multimodal question-answering processing module, configured to perform semantic association analysis between the question and the image frame and generate an answer result; The digital human interactive presentation module is used to drive the virtual digital human to output the answer results in the form of voice and synchronized animation.

[0015] Furthermore, the system also includes a mobile terminal and a cloud server; Among them, the mobile terminal performs image acquisition, question acquisition and key frame screening, and transmits the questions and screened image frames to the cloud server; The cloud server performs multimodal semantic fusion analysis and returns the answer results.

[0016] Furthermore, the multimodal question-answering processing module includes a visual encoder and a language encoder, which are respectively used to extract visual features of image frames and semantic features of question text, and fuse the visual features and semantic features through a cross-modal attention mechanism.

[0017] This approach leverages a large multimodal model to deeply integrate visual and verbal information, demonstrating a higher level of intelligence in understanding complex questions and scenarios. Compared to existing systems that prioritize visual recognition over verbal responses, this approach extracts visual evidence and infers directly based on the semantics of the question, avoiding information loss and misunderstanding during transmission. Furthermore, because the cloud-based model can integrate vast amounts of knowledge, the answers provided by the digital human are rich and explanatory, with additional explanations when necessary, enhancing the informative and engaging nature of the interaction.

[0018] Furthermore, the system also includes an abnormality monitoring and processing module for real-time monitoring of network connection status and image quality. When abnormal network delay or image distortion is detected, it triggers dynamic optimization of video acquisition strategy and inference process.

[0019] In a third aspect, the present invention also provides a computer device, comprising a management platform and a memory, wherein the management platform is connected to the memory, the memory is used to store computer programs, and the management platform is used to execute the computer programs stored in the memory, so that the computer device executes the aforementioned method of calling a camera to realize real-time two-way visual interaction of digital humans.

[0020] In a fourth aspect, the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a management platform, the method of calling a camera to realize real-time two-way visual interaction of a digital human is implemented.

[0021] The present invention uses a mobile terminal camera to capture a real-time video stream of the environment. Based on the semantic content of the user's question, a heuristic scoring function is used to perform keyframe screening, extracting only image frames or their features that are highly relevant to the question. The screened visual data and the user's question are transmitted to the cloud, where a visual encoder extracts high-level image features and a language encoder generates text features. A cross-modal attention mechanism is used to achieve feature alignment and joint reasoning to generate the answer text. The digital human outputs the answer text through speech synthesis and synchronized anthropomorphic animation, and combines spatialized feedback to form an interactive closed loop. The system controls end-to-end latency to an extremely low range through the coordinated optimization of lightweight terminal preprocessing and deep semantic fusion in the cloud. At the same time, it relies on a multimodal feedback mechanism to enhance interactive immersion, significantly improving the digital human's real-time environmental perception and natural interaction capabilities in scenarios such as intelligent customer service and AR navigation.

[0022] Beneficial effects By implementing the above-mentioned method and system for realizing real-time two-way visual interaction of digital humans by calling a camera provided by the present invention, the following technical effects are achieved: (1) This application significantly shortens the response time from user questions to digital human responses through terminal-side key frame screening and end-cloud collaborative processing. In a typical 4G / 5G network environment, it can achieve nearly instant feedback. Compared with the traditional approach of uploading the entire video to the cloud for analysis, this method only needs to transmit and process less than half the number of frames to return the answer. Even when conducting complex visual question-and-answer sessions on mobile devices, the interaction process is almost free of noticeable lag or long waits. The digital human can answer questions as they are asked, greatly improving the smoothness and naturalness of the interaction.

[0023] (2) Reduce the burden of device computing and network communication through multi-level optimization. On the one hand, the terminal's lightweight local processing avoids uploading large amounts of raw video data to the cloud. Instead, it extracts key information before transmitting it, reducing the amount of data transmission exponentially. On the other hand, the terminal only performs lightweight operations, leaving heavy computing tasks such as deep neural network inference to the cloud, saving local CPU / GPU computing power and electricity. Since the data sent is more streamlined, the network bandwidth occupancy and user traffic consumption are also smaller, making the system more economical and efficient in a wireless network environment. While meeting real-time requirements, it achieves a balance between low resource usage and high-performance response, adapting to the deployment requirements of mobile or embedded devices.

[0024] (3) A cross-modal large model is used to achieve a deep integration of visual information and language information, which has a higher level of intelligence in understanding complex questions and scenes. End-to-end cross-modal fusion enables the system to handle various types of visual questions and answers. Compared with the existing separate solution of "visual recognition first and then language answer", the present invention directly extracts visual evidence based on the semantics of the question and conducts joint reasoning, avoiding loss and misunderstanding in the process of information transmission. In addition, because the cloud model can integrate massive amounts of knowledge, the answers given by the digital human in this method are rich in content and explanatory, and additional explanations will be given when necessary, which improves the knowledge and fun of the interaction.

[0025] (4) The present invention creates a simulated "face-to-face" interactive experience. Users are no longer limited to preset instructions or button operations, but can communicate freely with digital humans through natural language and body movements; the digital humans respond in an anthropomorphic manner, and the degree of interactive freedom and immersion far exceed traditional human-computer interaction methods. In addition, digital humans give communication emotional warmth through rich facial expressions, body movements, and changes in tone. Compared with a pure text question-and-answer interface, this system can effectively improve user satisfaction, and the average single-time usage time and repeat usage rate of users have also increased significantly. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to make the above-mentioned method and system of calling the camera to realize real-time two-way visual interaction of digital humans more obvious and easy to understand, the following is a brief introduction to the drawings required for use in the specific implementation of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 This is a flow chart of the real-time two-way visual interaction method for digital humans according to the present invention; Figure 2 This is a schematic diagram of the end-cloud collaborative architecture of the digital human real-time two-way visual interaction system of the present invention; Figure 3It is a structural diagram of the visual language multimodal semantic fusion model described in the present invention. DETAILED DESCRIPTION

[0028] Example 1: A method for using a camera to realize real-time two-way visual interaction of digital humans is provided. The method flow is as follows: Figure 1 Shown, including: Step 1: The terminal device uses the camera to capture a high-speed, real-time video stream of the user and their surroundings. The camera acquisition module dynamically adjusts the frame rate and resolution based on the current network conditions and terminal computing resources, and performs basic preprocessing on the captured raw video frames to ensure stability and image quality for subsequent analysis.

[0029] Step 2: When the user asks a question in voice and / or text, the system captures the question and associates it with the current visual content captured simultaneously within the camera's field of view. For voice questions, the system first transcribes the speech into text in real time through automatic speech recognition, thereby obtaining the text corresponding to the visual content. This step semantically binds the user's natural language question to the camera image, ensuring that subsequent processing focuses on the visual area of ​​interest to the user.

[0030] Step 3: The visual perception module on the terminal side intelligently screens and preliminarily identifies the real-time video frames to filter out redundant information frames and extract key information frames that are highly relevant to the semantics of the question. On the one hand, the module uses a heuristic algorithm to select a subset of frames that may contain answer clues from consecutive frames; on the other hand, it performs necessary lightweight recognition on the selected frames. Through this key frame intelligent extraction strategy, the terminal only needs to process and upload a very small number of key frames to obtain the visual basis required to answer the question, avoiding uploading the entire video stream to the cloud for processing, thereby significantly reducing communication bandwidth usage and terminal computing load. After completing the key frame extraction and preliminarily identifying, the local module sends the selected key frame image or extracted visual features to the network communication module to proceed to the next step.

[0031] Step 4: The terminal sends the user's question text and the key image frames selected locally to the question-answering processing module in the cloud through the network communication module. After receiving the data, the cloud uses the visual language multimodal semantic understanding model to perform cross-modal fusion reasoning on the image and text: on the one hand, the cloud's visual encoder extracts the high-level semantic features of the image; on the other hand, the language encoder encodes the question text into a text feature vector. The features of these two modalities are then mapped to a unified high-dimensional semantic space for deep joint reasoning and analysis, and finally the answer text to the question is generated. After the cloud completes the answer reasoning, it immediately sends the result back to the terminal device via the network. Because the terminal pre-filters and compresses the uploaded data, the cloud can use its powerful computing power to complete complex reasoning with lower bandwidth usage. The end-cloud collaboration ensures high accuracy and low latency in question and answer.

[0032] Step 5: The digital human interactive presentation module on the terminal receives the answer text returned by the cloud and provides feedback to the user in an anthropomorphic, multimodal manner. The specific process includes: using text-to-speech synthesis technology to convert the answer text into natural and fluent speech; driving the digital human's virtual image to generate lip movements and facial expressions synchronized with the speech in real time to announce the answer; and, when necessary, combining appropriate body movements or visual aids to enhance the feedback effect. For example, when the answer involves a physical object in the camera image, the digital human will point to the object or highlight the relevant area on the real-time image on the terminal screen and overlay text descriptions.

[0033] A system for implementing real-time two-way visual interaction of digital humans by calling a camera is also provided, the system comprising: The camera acquisition module, deployed in mobile terminals, uses the device's camera to capture real-time video streams or still images. This module supports a variety of camera types and resolutions, dynamically adjusting the acquisition frame rate based on network and terminal computing resources to balance image clarity with real-time interaction. The camera acquisition module performs preliminary image processing on captured video frames and then provides the preprocessed frame data to the local visual processing module. This module acts as a visual sensor for the digital human, continuously providing the system with a high-quality source of environmental visual information.

[0034] The visual perception and key frame screening module is deployed on the terminal device to perform real-time lightweight analysis and processing on the raw image data captured by the camera, including but not limited to image compression and downsampling, basic target detection and recognition, and intelligent key frame extraction algorithms. This module introduces a language semantics-guided key frame screening strategy that can intelligently filter video frames based on semantic clues in user questions to extract key information. Specifically, the local module uses a language semantics-driven heuristic scoring function to score and sort frame sequences, giving priority to frames with higher scores, skipping redundant frames, and stopping further frame analysis in time when it is determined that sufficient visual key information has been collected, and outputting the required visual feature results. The key frame scoring function is formally expressed as:

[0035] Where, For the Key frame scoring of frame images; It is a heuristic scoring function that calculates the score based on the degree of match between image features and question semantics; For the Frame Image Functions for extracting visual features; Text to ask the user questions Function for semantic encoding. Through the intelligent frame selection mechanism, the module can adaptively select pictures that are highly relevant to the semantics of the question and answer, greatly reducing the amount of data that needs to be processed and transmitted subsequently, while ensuring that no key information related to the question is missed. During the local visual recognition process, the module also calls the AI ​​acceleration hardware of the terminal device to execute a lightweight pre-training model to speed up processing and enrich the recognition dimension. It should be noted that the basic visual recognition technology used by this module can adopt existing mature algorithms, and the implementation method is not limited to a specific model. The module's key frame intelligent extraction and semantic relevance filtering mechanism ensure that the system can correctly understand the semantic focus of the user's question in the current scenario even if it only processes a very small number of frames.

[0036] The network communication module manages data transmission and interaction between the terminal and the cloud server, ensuring that locally processed critical information is uploaded securely, efficiently, and with low latency to the cloud, and that the results returned by the cloud are promptly delivered to the terminal interface. The communication module is preferably implemented using a low-latency asynchronous request-response mechanism based on HTTP / HTTPS or WebSocket protocols. Regarding data transmission, the module packages the user's question text and selected keyframe images together, performing necessary encryption and compression before uploading to reduce bandwidth usage. Larger image data is further compressed using formats such as JPEG and WebP, or only high-level semantic features extracted through encoding are sent to conserve bandwidth. To maximize network round-trip time, the communication module operates in a non-blocking manner. While awaiting a response from the cloud, the terminal can perform subsequent preparations in parallel, maximizing communication latency. This module places particular emphasis on real-time optimization and data security, implementing a jitter-resistant transmission strategy to minimize the impact of network latency and implementing necessary encryption measures throughout data transmission to prevent privacy leaks.

[0037] The cloud-based question-answering processing module is deployed on a cloud server. After receiving the user's question text and related visual data from the terminal, this module first maps the image features and text features into a unified representation space through a multimodal semantic fusion model, achieving cross-modal understanding and alignment. In a preferred embodiment, the cloud-based module loads a specially trained visual question-answering model, which contains a visual attention mechanism that dynamically focuses on relevant areas in the input image based on the question, thereby accurately capturing the relationship between the image semantics and the question intent. Once the multimodal model completes semantic parsing by fusing the question text and image information, the internal generative reasoning unit generates the natural language text of the answer. During the answer generation process, the cloud-based module also calls the knowledge base or searches the network as needed to supplement additional information. The entire cloud-based reasoning process is optimized to ensure accuracy while reducing response latency. For example, a hierarchical reasoning strategy is adopted, where a smaller model first quickly provides a preliminary answer, and then a larger model fine-tunes the answer to improve the quality of answers to complex questions; or the model is pruned and quantized to increase reasoning speed. In multi-round dialogue scenarios, the cloud module also supports context association and result caching mechanisms. For users' continuous related questions, the previous dialogue content is used as the context input model, or recent processing results are cached to avoid repeated reasoning, thereby improving the coherence of the dialogue and interaction efficiency.

[0038] During the model training phase, the system uses a cross-modal semantic alignment contrastive learning strategy to optimize the fusion representation of visual and text features, enabling the model to better understand both image and language information. For example, correctly matched image-text pairs are brought closer together, while irrelevant image-text pairs are pushed away. Specifically, assuming that there is a set of image feature vectors and a set of corresponding text feature vectors in the training batch, a symmetric cross-modal contrastive loss function is defined to bring matching image-text pairs closer together and distinguish mismatched pairs:

[0039] Where, is the cross-modal contrast loss value; is the number of samples in the training batch; For the The visual feature vector of the image; For images Matched question text feature vector; is the image feature With text features The cosine similarity between is the temperature coefficient; By minimizing this loss function, we can shorten the distance between matching image-text pairs in the semantic space and separate mismatched image-text pairs, thereby significantly improving the model's ability to align and understand across visual and language modalities.

[0040] The digital human interactive presentation module, deployed on the terminal device, presents the answers returned from the cloud to the user in an anthropomorphic, multimodal manner. This module includes components such as a speech synthesis unit, a virtual human rendering unit, and a multimodal feedback mechanism. The speech synthesis unit uses text-to-speech technology to convert the answer text into natural, fluent speech. The virtual human rendering unit drives a pre-built digital human model to generate lip movements and facial expression animations synchronized with the speech, ensuring that the lip movements are strictly consistent between the sound and the image. Preferably, the digital human avatar adopts facial expressions or body movements appropriate to the content and tone of the answer, such as a smile when answering a pleasant question or a solemn expression when explaining a serious topic. If the answer involves pointing to an object, the digital human will gesture toward the object or turn its head to gaze in the corresponding direction. This module also provides visual feedback. When the digital human's answer closely relates to the live camera feed, the system interface displays relevant images or text descriptions, highlighting the area of ​​the image of interest to the user. For example, if a user asks the camera, "What kind of flower is this?" and there's a flower in the picture, the digital human will respond with "It's a red rose." The flower will be marked with a box and the word "rose" will be annotated on the real-time video screen in the interface, creating an intuitive interaction with both pictures and text. This dual feedback design, combining language and vision, allows users to not only hear the answer but also directly see the basis for the answer, greatly enhancing the immersion and accuracy of the interactive experience.

[0041] Through a modular design, the aforementioned modules feature clear division of labor and close collaboration, forming a complete end-cloud collaborative system architecture. On the one hand, standardized data interfaces are used for interaction between modules, supporting loose coupling and flexible expansion. Developers can replace or upgrade a module without modifying the implementation of other modules. On the other hand, modules are distributed and deployed on the terminal or cloud based on performance requirements, forming a real-time interactive system for end-cloud collaboration. Computationally intensive cross-modal reasoning tasks are executed in the cloud, while tasks with high real-time requirements, such as camera acquisition and rendering feedback, are retained on the terminal. The local and cloud interfaces are connected via high-speed network interfaces, allowing the system to leverage the computing power of the cloud while minimizing the impact of network latency. This architectural design effectively integrates the various elements of camera visual perception and digital human interaction, providing a solid and flexible technical foundation for the large-scale application of digital human technology.

[0042] The key algorithm mechanism adopted by the present invention, including functions and their principles, is as follows: Image clarity scoring function: Image clarity is measured by calculating the gradient magnitude energy of the image. Represents a grayscale image in pixels The intensity value at , the clarity score of the frame It can be defined as the global average gradient energy:

[0043] Where, Rate image clarity; and are the height and width of the image in pixels respectively; and For the image The formula actually calculates the average value of the edge gradient strength in the entire image: the clearer the image, the more distinct the internal edge contours, the larger the gradient amplitude, and the higher the score; conversely, if the image is blurred, the overall gradient value is smaller and the score is also lower. The system is set with a clarity threshold. When the clarity of a frame is detected to be lower than the threshold, the frame is considered to be of poor quality, and the local module discards it and does not send it to the cloud as a key frame; if multiple frames are blurred continuously, the user is prompted to adjust the camera focus or increase the ambient light to ensure that the frame image used for analysis is clear and usable.

[0044] Image-language joint attention weight calculation: In the cloud multimodal model, a cross-modal attention mechanism is introduced to measure the relevance between images and text. This system uses an attention model that interacts between visual transformers and text transformers. Assuming that the visual encoder extracts the image Regional feature vector , the text encoder generates the question sentence latent feature representation . For the text features, image features The attention weight corresponding to the regional features Defined as:

[0045] Where, For the The text feature pair The attention weight of each image region; The question text Semantic feature vector of each word / position; For the The visual feature vector of the image; is the total number of regions into which the image is divided; is an exponential function used to normalize the weights. The above weights constitute a normalized probability distribution. , reflecting the attention distribution of each part of the text question on different image regions. Based on these attention weights, the model locates the area in the image that is most relevant to the question semantics, thereby improving the accuracy of cross-modal understanding. For example, the matching degree score between the entire image and the entire question text is further calculated: the attention weight of each position of the text on the image is applied to the corresponding image region features and the sum is obtained to obtain a fusion representation, and then the average value of the similarity is calculated with the overall text features to obtain a scalar value that measures the degree of semantic matching between the image and the text. The higher the value, the more semantically consistent the image content is with the given question. It should be noted that the above-mentioned attention calculation is usually completed in one go in the form of matrix operations within the model, but its effect is equivalent to calculating and normalizing the weights element by element according to the above formula. Using this cross-modal attention mechanism, the model can dynamically focus on the area in the image that is most relevant to the question, thereby more accurately capturing the relationship between visual semantics and question intent.

[0046] Model output confidence assessment mechanism: In order to evaluate the credibility of the answers given by the cloud-based question answering model, a confidence scoring method for the model output is introduced. Usually, the confidence is determined by the probability distribution of the model when generating the answer. For example, if we regard visual question answering as a The classification problem of candidate answers, the model outputs a score for each candidate , then the model's confidence in the final predicted answer Defined as the Softmax probability corresponding to this option:

[0047] Where, is the confidence of the model in the predicted answer; is the logit value of the predicted answer; is the total number of candidate answers; For the The logit value of the candidate answer.

[0048] If the model outputs free-text answers of indefinite length in a generative manner, confidence is calculated in a similar manner. For example, the entire generated answer is considered as a series of word joint events, and the probability of the model outputting this sequence is used as the confidence score of the answer. The system sets a confidence threshold. When the model's confidence in a certain answer is lower than the threshold, the preset backup strategy is triggered to avoid giving misleading replies when the model is not certain enough. Through the above-mentioned confidence assessment mechanism, the system obtains a credibility indicator at the same time as generating the answer, which is used by subsequent modules to decide how to present the answer. When the confidence is high, the answer is given directly. When the confidence is low, the digital human's tone and expression will be appropriately adjusted, or an uncertainty prompt will be given, thereby improving the reliability of the interaction and user trust.

[0049] The diagram of the end-cloud collaborative architecture is as follows Figure 2 As shown, the data flow and time sequence of each module working together under normal circumstances are as follows: When a user launches the Digital Human app on a mobile device, the camera begins continuously capturing video of the environment. A typical interaction scenario involves the user pointing the camera at a scene or object while simultaneously asking the Digital Human a question via voice or text. Only when the system detects that the user has posed a question does the local visual perception module perform the aforementioned keyframe extraction and preliminary recognition on the current video frame sequence, refining the image content relevant to the question in real time. For example, if the user asks, "What flower is this?" the system immediately captures the frame at the time of the question and quickly identifies the primary flower as the target of interest locally on the device. Alternatively, if the user asks, "What is he doing?" the local module selects several representative frames from the continuous video, capturing key moments of the character's movements to facilitate understanding. The network communication module then packages the user's question text and the selected keyframe data and sends it to the cloud.

[0050] The architecture of the visual language multimodal model is as follows Figure 3 As shown, upon receiving a request, the cloud-based question-answering processing module first extracts visual features from the image using a visual encoder. These features, along with the textual question, are fed into a multimodal model for joint encoding and reasoning. The model internally uses mechanisms such as self-attention and cross-modal attention to understand the relationship between image and textual information and generate an answer. During this generation process, the model automatically accesses its knowledge base or performs online searches to identify flower varieties or interpret the meaning of a person's actions, if necessary. This complex reasoning process is transparent to the end user. Once the cloud generates the textual answer, it immediately sends the result back to the end device via the network.

[0051] After receiving the answer text, the digital human interactive presentation module on the terminal immediately invokes the text-to-speech engine to synthesize the corresponding speech. It then drives the virtual digital human's facial and motion models to synchronize its lip movements with the speech, announcing the answer to the user in real time. For example, when the digital human answers, "This is a red rose," the virtual human in the application interface will turn its head and gaze toward the flower in the camera view, pointing at it. Alternatively, it will directly select the flower's location and label its name on the live camera view. The entire process, from the user asking the question to the digital human's response, typically takes only a few seconds or even less, achieving a near-real-time interactive experience.

[0052] The present invention achieves high-performance real-time visual question and answer through seamless collaboration between the terminal and the cloud. The terminal fully filters and refines the information before uploading it, which reduces the computing load and communication bandwidth pressure on the cloud; the cloud exerts its powerful deep learning reasoning ability to solve the core problem of visual semantic understanding. It is this architectural design that allows the user experience to remain smooth and natural even when running complex visual question and answer applications on mobile devices with limited computing and battery resources. This method and system effectively overcome the difficulties of real-time interaction of vision and language fusion in the existing technology, has high real-time performance, high accuracy and good user immersion experience, can be widely used in various scenarios such as intelligent customer service, virtual assistants, and distance education, and has significant practical value and promotion prospects.

[0053] Given that digital human applications need to be widely deployed on different platforms, such as mobile devices and computer web pages, a unified adaptation plan and verification mechanism have been developed to address the differences between platforms to ensure cross-platform consistency and stable performance of system functions. The main measures include: At the architectural design level, each submodule follows platform-independent interface specifications, encapsulating and shielding the underlying differences between different systems. For example, the camera acquisition module internally calls different native APIs for different platforms, but provides a unified frame data format and frame rate control interface to the upper layer, thereby ensuring consistent behavior of the camera acquisition function on different platforms. For another example, the virtual human rendering module is implemented based on a cross-platform graphics rendering engine, using Metal or OpenGL ES interfaces on mobile devices and WebGL or WebAssembly technology on the web. Through good encapsulation, the rendering and interactive control interfaces of the digital human image remain consistent. Through these module encapsulations, developers do not need to pay attention to platform details, and each module can run equivalently on different systems.

[0054] The terminal-cloud collaborative architecture is naturally compatible with devices of different performance levels. By dynamically adjusting the division of tasks between the terminal and the cloud, a nearly consistent response speed is achieved across all platforms. On high-performance terminals, local computing power is used as much as possible to handle more real-time recognition tasks. On low-performance terminals or in web browser environments, cloud computing power is relied upon more, and the terminal only performs lightweight work. The system automatically adjusts the depth of terminal-side processing based on device performance and current network conditions. For example, when network bandwidth is limited or terminal performance is insufficient, the frame rate or resolution of camera video capture is reduced, and only necessary features are extracted and uploaded to reduce data transmission and computational latency. Conversely, when the network is good and terminal performance allows, video quality is improved or local processing is increased to obtain richer semantic information. This adaptive strategy ensures that regardless of the device used by the user, the system's response time remains within an acceptable range in various network environments, achieving a balanced and stable real-time interactive experience across all platforms.

[0055] In terms of user interface and interaction logic, unified design specifications and adaptation solutions are adopted to ensure that the layout of the digital human interface, the logic of virtual human behavior, and the user interaction method are consistent across all platforms. During the development process, Apple applications, Android applications, and Web applications are synchronously tested through a cross-platform testing framework to compare the display effects and response delays on different operating systems and devices to ensure that there are no problems with missing functions or inconsistent experiences. If a platform is found to have special compatibility issues, the system will add a special adaptation branch to the module implementation of that platform for optimization, while keeping the main process logic unchanged. In summary, whether the user uses an Apple or Android mobile device, or a computer browser to access, they can obtain a digital human visual interaction experience with equivalent functions, smooth performance, and a unified interface.

[0056] Aiming at various abnormal situations that may occur during real-time two-way visual interaction, a comprehensive abnormality detection and processing mechanism is designed, covering various links such as network communication, image acquisition quality, cloud inference process, terminal operation status, etc., to ensure that the system can still operate stably in complex environments. It mainly includes: During communication between the terminal and the cloud, if an unstable network connection or excessive round-trip latency is detected, the communication module will initiate a fault-tolerance strategy. First, a timeout retransmission mechanism is implemented. If no response is received from the cloud within a preset time after a request is sent, the request or key data frame is automatically retransmitted, ensuring that the request can be redelivered in the event of occasional packet loss. Second, for persistent high latency or disconnections, the system supports rapid network channel switching or service degradation strategies. For example, it automatically switches from WiFi to a cellular network, or reduces the frequency and resolution of video frame transmission to alleviate network pressure when bandwidth is insufficient. If the network interruption lasts too long and cannot be restored in time, the system will promptly inform the user of the current network anomaly and temporarily enter offline waiting mode, automatically continuing the unfinished Q&A process after the connection is restored. Through these measures, even in the face of network fluctuations, the system strives to ensure the completion of the Q&A task, or at least to provide prompts when problems arise rather than crashing directly.

[0057] To address potential quality issues with image frames captured by the camera, a quality detection and adaptive adjustment mechanism is integrated into the local visual processing module. On the one hand, the system calculates metrics such as clarity and brightness for each frame and compares them with preset thresholds. If the current frame is found to be too blurry or severely insufficiently bright, its reliability is determined to be low and it is not used as a keyframe to be sent to the cloud. If multiple consecutive frames do not meet the standards, the terminal interface will prompt the user to adjust the camera or shooting environment. On the other hand, when the camera acquisition module detects poor frame quality, it automatically adjusts the camera parameters to improve the image effect, such as enabling autofocus to refocus and enabling automatic exposure compensation to increase brightness. If certain frame data is severely distorted or damaged, the system will discard the abnormal frame and wait for the next frame data to avoid passing the abnormal data to subsequent modules for analysis. Through these strategies, the quality of image frames entering the question and answer analysis process is ensured to meet standards, thereby improving the accuracy of visual understanding.

[0058] In the cloud-based question-answering processing module, timeout monitoring and result verification mechanisms are implemented to address the potential for response timeouts or unreliable results from large multimodal models. The cloud starts a timer when processing a request. If the inference time exceeds a preset threshold, it is considered a possible timeout. In this case, the system has two response strategies: First, abort the inference process of the current large model and use a backup lightweight model or a preset answer as a fallback response, such as a polite "unable to answer" prompt to ensure timely response to the user. Second, continue waiting for the large model to complete inference while providing progress feedback to the user interface, presenting the answer immediately after the result is generated. In addition to timeout monitoring, the cloud-based module also verifies the validity of the answer returned by the model, such as checking whether the answer is empty or whether it clearly does not match the query's semantics. If the answer is invalid, the system logs the exception and attempts to re-invoke the model generation or retry the solution using a different inference path. If multiple retries fail, the cloud-based module eventually returns an error flag, and the terminal will notify the user of the error in a friendly manner or guide the user to try again later. The above timeout and result verification processing avoids long periods of no response or irrelevant answers due to model anomalies, ensuring the robustness of the interaction process.

[0059] On the device side, the small AI models that the local vision processing module relies on may experience anomalies such as loading failures, initialization errors, or runtime crashes. Remediation strategies have been designed accordingly. Upon detecting that a device AI model cannot be loaded or invoked properly, the system immediately switches to a more downgraded operating mode, minimizing the downgrade without sacrificing core functionality. The preferred approach is to bypass the faulty module and simplify the process, for example, skipping the local text recognition step and sending image frames directly to the cloud for text recognition to ensure uninterrupted question-and-answer processing. If the faulty module's functionality is essential to the overall system, a backup cloud service is used to replace the device model. While this may increase latency, it maintains functional integrity. For example, if the device's facial recognition model is unavailable, key frames can be uploaded to the cloud for cloud-based recognition assistance. Simultaneously, the device logs the error and notifies the application management module for subsequent remediation. If anomalies in the device's AI inference results are detected during operation, the system implements an isolation and downgrade strategy: If necessary, complex device analysis is temporarily disabled, retaining only the core functions of camera frame forwarding and cloud-based question-and-answering. The user is then notified of insufficient device performance or temporary unavailability of the corresponding function. Through the above measures, the present invention minimizes the impact of terminal AI module abnormalities on the system, ensuring that even if some local intelligent functions fail, the main question-and-answer interaction of the digital human can still basically operate without interrupting the user experience.

[0060] In summary, the present invention covers potential abnormal situations in various aspects, from network communication and image quality to model reasoning and device status, through the design of a comprehensive exception handling mechanism, and provides comprehensive and effective response measures to ensure that the system can still run stably and reliably under various complex conditions, providing users with a continuously good interactive experience.

[0061] Example 2: This example demonstrates the system's application in a real-time video conversation scenario. A user launches a digital human application with an anthropomorphic avatar on their mobile device and interacts with it similar to a video call. The terminal's front-facing camera continuously captures the user's facial expressions and surroundings, providing this information to the digital human system. The user then asks the digital human questions through voice. The digital human "hears" the user's questions while "seeing" the user's surroundings through the camera, providing responses tailored to the situation.

[0062] During the interaction process, the camera acquisition module continuously captures frames of the user and the environment. The local visual processing module analyzes these frames in real time, identifying the user's facial expressions, body movements, and objects in the background scene. When a user asks a question, the system first transcribes the speech into text using automatic speech recognition. Then, the terminal performs local video frame filtering. For example, it selects the most recent frame with the clearest portrait on the wall behind the user and detects the face of the person in the portrait in that frame. The terminal then sends the processed question text along with the image frame to the cloud. Upon receiving the question, the cloud-based multimodal model uses the facial recognition sub-model to identify the likely person in the painting and, based on the semantics of the question, understands the identity of the person in the portrait. If the portrait is identified as a historical figure, the model retrieves information about the person from its own knowledge base or online encyclopedias, ultimately generating a textual answer. After the answer is transmitted back to the terminal, the digital human interface provides feedback to the user in an anthropomorphic manner, announcing the information through voice, while the virtual human turns its head to look in the direction of the portrait behind the user and gestures to indicate it. The user thus personally felt that the digital human had truly "seen" the painting and correctly identified the figures within, enabling highly natural contextual question-and-answer interaction between the human and the digital human. Using camera visual input, the system enabled the digital human to recognize the user's physical surroundings and understand their relevance to the user's questions. Collaborative processing between the terminal and the cloud ensured real-time recognition and Q&A. The digital human provided responses through natural expressions and body movements, strengthening the user's confidence in its accuracy and their willingness to continue communicating.

[0063] Example 3: This embodiment introduces the application of the system in daily life scenarios, namely, a digital human as a personal intelligent assistant. Users can have real-time visual communication with the digital human assistant on mobile phones, tablets, and other terminals. Typical application scenarios include: While traveling, users can use their camera to capture their surroundings and ask questions. The digital human assistant can then identify landmarks or historical sites in the frame and provide real-time historical context and information. For example, if a user points to a statue and asks, "Who is this?" the digital human will identify the person depicted, answer their name and life story, and point to the statue with virtual gestures.

[0064] While browsing a store, users scan items with their cameras and ask the digital human for information or to find out if there's a better deal. The digital human uses text recognition and other technologies to read the text and logos on the product packaging, identify the brand and model, and instantly access price comparisons online. It then provides the user with the product's online price or recommends alternatives. Throughout this process, the digital human uses vision to capture product barcodes and packaging information, enabling intelligent analysis of the items it sees and providing personalized shopping recommendations.

[0065] When users encounter device connectivity or usage issues, they simply point their camera at the device in question and describe the symptoms to the Digital Human. The Digital Human observes visual information such as the indicator lights and interface connections on the device panel, and, based on the user's verbal description, provides troubleshooting and operational guidance. For example, when a computer is unable to connect to the internet, the user uses a camera to capture the router's status indicator lights. The Digital Human identifies the flashing pattern and suggests possible causes. Through its visual understanding of the environment, the Digital Human Assistant significantly improves the efficiency and ease of use of remote technical support.

[0066] Through the above scenario applications, the digital human assistant can understand the user's real situation, and then provide what you see is what you get service combined with voice dialogue, becoming a caring and intelligent consultant for the user.

[0067] Example 4: This embodiment applies the system to a home environment, acting as a smart home butler. A camera network is deployed in the home, and the digital human butler uses the cameras to monitor the home situation in real time and communicate with the residents. Specifically, it includes: When an unusual situation occurs at home, the Digital Human Butler automatically calls up relevant cameras to check the scene. If the camera feed shows only pet activity, the Digital Human uses a speaker to notify the owner, warning them not to worry. If it detects a suspected intruder, the Digital Human immediately issues a warning and triggers the alarm system, while also recording a video feed for evidence. If an elderly person at home alone detects a fall, the Digital Human can detect it through vision and immediately initiate a voice inquiry, automatically calling emergency services if no response is received. Empowered by vision, the Digital Human Butler maintains 24 / 7 vigilance to protect the family's safety.

[0068] Family members can communicate directly with the digital butler through a central control device with a screen. For example, they can ask the digital butler on the screen, "What's in the refrigerator?" and simultaneously share the camera feed inside the refrigerator with the digital butler. The digital butler will check the ingredients in the feed and answer the question about the refrigerator's inventory, and further offer recipe suggestions. For example, if the owner asks, "Are all the windows closed?" before leaving, the digital butler will retrieve the cameras in each room to confirm the status of each window. Using voice commands and highlighting the corresponding camera feed on the tablet interface, the digital butler will inform the owner of the closure status of each window, thus preventing safety hazards.

[0069] The Digital Human can also monitor family members' emotional well-being by recognizing facial expressions and body language through cameras. For example, if a senior living alone is detected to be unusually silent or depressed, the Digital Human will proactively appear on the home screen to chat and comfort them, or remind their family members to pay more attention. When a child comes home from school, if the Digital Human detects a positive mood, it will smile and interact, offering warm and caring responses like a family member.

[0070] The application of the system in smart home scenarios enables the intelligent assistant to not only understand the instructions of residents, but also understand environmental changes and the status of users, thereby providing more humane, safe and reliable home services.

[0071] Example 5: This embodiment applies the system to guide tours in public places and commercial shopping guide services: In museums, art galleries, and other venues, digital human guides are displayed on large screens. Cameras installed in the venues capture the visitor's location and behavior in real time. When a visitor approaches an exhibit and pauses to look, the digital human guide "walks" in the direction of the exhibit they are looking at and begins explaining the background story of the exhibit. If the camera detects a confused expression or signs of a question, the digital human will adaptively adjust the depth and speed of the explanation, providing targeted explanations until the visitor expresses understanding or nods. Throughout this process, the digital human visually understands the visitor's attention and reactions, greatly enhancing the visitor experience.

[0072] In retail stores or trade shows, digital humans act as virtual shopping guides. In-store cameras identify in real time the number of customers entering the store, the areas they visit, and the items they are interested in. If a customer gazes at a particular item for an extended period, the digital human shopping guide will proactively greet them and explain the product's features. If the camera sees a customer repeatedly trying to size a piece of clothing, the digital human will proactively suggest a try-on or provide stock inquiries. Multilingual capabilities allow digital human shopping guides to seamlessly serve customers from different countries. Through visual interaction, the shopping guide service becomes more proactive, attentive, and personalized, thereby increasing business conversion rates.

[0073] In these scenarios, the system empowers virtual tour guides and shopping guides with contextual awareness, proactively detecting user needs and providing targeted services. These digital human service personnel will significantly enhance the intelligent service levels and customer satisfaction of venues and stores.

[0074] Example 6: This embodiment applies the system to the field of education, empowering digital human teachers or coaches to achieve two-way interaction in remote teaching and training: In online classroom scenarios, a digital human teacher appears as a virtual avatar on each student's screen, delivering a live lecture. Using student cameras, the system allows the teacher to observe each student's attention in real time. For example, if the digital human observes a student frowning frequently and determines they may be struggling to follow the lesson, it will temporarily slow down the lecture or call on them to ensure their understanding. When a student raises their hand, the digital human will turn its gaze to them, call on their name, and patiently answer their questions. If the majority of students appear confused, the digital human will adjust its teaching strategy and provide more examples to help them understand. After class, students can show their homework to the digital human teaching assistant via webcam. The digital human automatically recognizes the content, corrects it, and points out errors through voice guidance and on-screen annotations. For schools in remote areas with limited teaching staff, a single digital human teacher can be deployed to teach students simultaneously across multiple locations via a large network screen, integrating webcam interaction to provide personalized instruction. During vocational skills training, the digital human coach observes students' actual operating techniques through webcams, pointing out irregularities in real time and demonstrating corrective actions, providing personalized guidance and correction.

[0075] Digital humans can also serve as rehabilitation training instructors or psychological counseling assistants. Through camera video, digital human rehabilitation coaches supervise patients completing prescribed rehabilitation movements at home, identify whether their body postures are in place, and immediately correct any deviations through voice and demonstrate the correct posture with a virtual image to ensure the quality of rehabilitation training and prevent secondary injuries caused by incorrect movements. In psychological counseling scenarios, the digital human assistant observes the patient's facial expressions and body language through the camera. If it detects that the other person is depressed or crying, it will gently comfort and encourage them; and dynamically adjust the rhythm and content of the conversation based on the patient's feedback, so that the patient feels understood and cared for. Because digital humans can provide around-the-clock protection and companionship, it helps to improve the continuity of psychological treatment and the patient's sense of security.

[0076] By introducing visual interaction, digital humans in the education and training field have achieved comprehensive awareness of learners' states. Like a real teacher, they can focus on each student's response and guide them accordingly, achieving more efficient and intuitive knowledge transfer and personalized tutoring. This capability is particularly valuable in scenarios where teacher resources are insufficient or remote teaching is necessary.

Claims

1. A real-time two-way visual interaction method for digital humans, characterized in that: include: Acquire image frames of the user and their environment in real time through the camera of the terminal device; Accepting natural language questions raised by users in voice and / or text form, and establishing semantic associations between the questions and real-time image frames; Performing visual language multimodal semantic fusion analysis based on the question and the associated image frame, and generating a corresponding answer result; The answer result is output by a virtual digital human through voice synthesis and synchronized animated expressions.

2. The method according to claim 1, wherein: Before establishing semantic association between the question and the real-time image frame, key frames are screened on the obtained image frame sequence based on the semantic content of the question, and the key frames are highly relevant to the semantics of the question.

3. The method according to claim 1, wherein: The multimodal semantic fusion analysis is achieved through collaboration between the terminal device and the cloud server; Among them, the terminal device performs preprocessing and feature extraction on the image frame, and transmits the extracted visual features and question text to the cloud server; the cloud server performs deep cross-modal reasoning based on the received visual features and question text to generate an answer result.

4. The method according to claim 3, wherein: During the semantic fusion analysis process, the visual features of the image frame are extracted through a visual encoder, the semantic features of the question text are extracted through a language encoder, and the visual features and semantic features are fused using a cross-modal attention mechanism to align the question semantics with the image content.

5. The method according to claim 1, wherein: When the virtual digital human outputs the answer result, an anthropomorphic feedback behavior is performed synchronously; The anthropomorphic feedback behavior includes adjusting the line of sight toward the user, indicating the target object in the image frame through virtual body movements, or superimposing a target mark on the real-time image.

6. The method according to claim 1, wherein: The method dynamically monitors the network connection status and image frame quality of the terminal; When it is detected that the network delay exceeds the preset threshold or the image clarity is below the standard, the video acquisition parameters and question-answering reasoning strategy are adaptively adjusted to ensure the real-time and stability of the interaction.

7. A real-time two-way visual interaction system for digital humans, characterized by: When the system is running, the method according to any one of claims 1 to 6 is executed: The system comprises: A camera acquisition module, used to capture image frames of the user and their environment in real time; A question acquisition module is used to receive user questions in voice and / or text form; A key frame screening module, configured to screen semantic key frames from the image frames based on the semantic content of the user's question; A multimodal question-answering processing module, configured to perform semantic fusion analysis on the question and the selected key-frame image frames, and generate an answer result; The digital human interactive presentation module is used to drive the virtual digital human to output the answer result through voice synthesis and synchronous animation.

8. The system according to claim 7, characterized in that: The system also includes a mobile terminal and a cloud server; The mobile terminal includes the camera acquisition module, the question acquisition module and the key frame screening module, and is used to transmit the question text and the screened image frames to the cloud server; The cloud server includes the multimodal question-answering processing module, and is used to perform cross-modal semantic fusion analysis based on the received data to generate an answer result and then return it to the mobile terminal.

9. The system according to claim 7, characterized in that: The multimodal question-answering processing module includes a visual encoder and a language encoder, which are respectively used to extract the visual features of the image frame and the semantic features of the question text; the multimodal question-answering processing module also includes a cross-modal reasoning unit, which is used to fuse the visual features and semantic features based on a state attention mechanism to generate the answer result.

10. The system according to claim 7, wherein: The system also includes an anomaly monitoring and processing module for real-time monitoring of network connection status and image frame quality. When network delay anomalies are detected or image clarity is lower than a predetermined threshold, it triggers the adjustment of video acquisition parameters and dynamic optimization of the question-answering reasoning process.

Images