Method and system for realizing real-time two-way visual interaction of digital human by calling camera

Abstract

The application discloses a kind of calling camera to realize digital person real-time bidirectional visual interaction method and system, belong to multimodal interaction technical field, the method includes: through mobile terminal camera real-time collection user and the video stream of environment, and synchronously obtain user natural language question;Based on the key of relevant video screening of question semantics;Utilize multimodal model to the question text and key frame image are carried out cross-modal semantic fusion analysis, generate corresponding answer result;By virtual digital person through voice synthesis and synchronized anthropomorphic animation output answer, realize the real-time graphic text question and answer interaction between digital person and user.The present application is aimed at the problem that lack of visual context in existing digital person interaction, and a visual question and answer interaction scheme of end-cloud cooperation is proposed, which can effectively reduce the video semantic processing load of mobile terminal, realize the real-time graphic text question and answer interaction of end-cloud cooperation, greatly improve the perception ability of digital person to environment, and the intuitiveness and naturalness of user interaction experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal interaction, in particular to a method and system for calling a camera to realize real-time two-way visual interaction of a digital person. BACKGROUND

[0002] With the cross-fusion of artificial intelligence and virtual reality technology, digital people as multi-modal interaction carriers have been widely used in intelligent customer service, virtual tour and immersive education, etc. The existing system mainly realizes personification interaction through the integration of speech recognition, natural language processing and expression driving technology, but its perception ability is still limited to single-modal input, lacking real-time analysis ability of environmental visual information.

[0003] The current multi-modal system requires users to explicitly upload static images as input, and the system generates answers after offline analysis. Such a solution has two major defects: lack of time sequence continuity and interaction process fault. Although some research attempts to connect the camera to the interaction system, its function is limited to single instruction response. Mobile deployment faces three challenges: continuous video stream processing exceeds the terminal computing power bearing, original video upload causes network delay to surge, and the semantic coordination relationship between user questions and real-time pictures cannot be established.

[0004] In addition, the existing digital person output is mainly based on voice and basic expression animation, lacking spatial direction feedback, making it difficult for users to establish a cognitive mapping of "ask what you see".

[0005] Therefore, there is an urgent need for a low-latency, high-robustness two-way visual interaction mechanism on mobile terminals, which solves the problems of real-time, accuracy and immersion through end-to-cloud collaborative architecture and semantic-guided visual compression strategy. Based on the above problems, the present application proposes a method and system for calling a camera to realize real-time two-way visual interaction of a digital person. SUMMARY

[0006] PURPOSE OF THE INVENTION

[0007] In order to solve the above problems, the purpose of the present application is to provide a method and system for calling a camera to realize real-time two-way visual interaction of a digital person, which establishes the space-time association of visual flow and language interaction by realizing real-time visual analysis of the user's physical environment, designs a semantic-guided key frame selection mechanism and a cross-modal attention fusion model to reduce the computing load of mobile terminals and improve the accuracy of semantic understanding, and significantly improves the user's immersion and operation efficiency in different scenarios.

[0008] TECHNICAL SCHEME

[0009] To achieve the above object, the application provides a method and system for realizing real-time two-way visual interaction of digital people by calling a camera, which dynamically collects environmental video streams through a terminal camera, outputs time sequence images after adaptive frame rate control and preprocessing, performs key frame screening on image frames by using a heuristic scoring function based on the semantic content of user questions, and only retains a frame subset or visual features highly relevant to the semantic content of user questions; the terminal transmits the frame subset and user questions to the cloud, extracts image high-level features by a visual encoder and generates text features by a language encoder, and realizes feature alignment and joint reasoning through a cross-modal attention mechanism to generate an answer text; after receiving the answer text, the terminal drives a digital person to output an answer by speech synthesis and synchronous animation, and completes an interactive closed loop by combining spatialized feedback.

[0010] In the first aspect, the application provides a method for realizing real-time two-way visual interaction of digital people by calling a camera, which comprises the following steps:

[0011] Real-time acquisition of image frames of a user and his / her environment through a camera of a terminal device;

[0012] Acceptance of natural language questions in the form of voice and / or text from a user, and semantic association of the questions with real-time image frames;

[0013] Performance of visual language multi-modal semantic fusion analysis based on the questions and associated image frames, and generation of corresponding answer results;

[0014] Output of the answer results by a virtual digital person through speech synthesis and synchronous animation expression.

[0015] Further, before the semantic association of the questions with real-time image frames, key frame screening is performed on real-time collected frame sequences based on the semantic content of the questions, and the key frames are highly relevant to the semantic content of the questions.

[0016] Further, the multi-modal semantic fusion analysis is realized by cooperation of a terminal device and a cloud server;

[0017] In the method, the terminal device performs preprocessing and feature extraction on the image frames, and transmits the extracted visual features and question texts to the cloud server; the cloud server performs deep cross-modal reasoning based on the received visual features and question texts to generate answer results.

[0018] Further, in the semantic fusion analysis process, visual features of the image frames are extracted by a visual encoder, semantic features of the question texts are extracted by a language encoder, and the visual features and semantic features are fused by a cross-modal attention mechanism.

[0019] Further, when the virtual digital person outputs the answer results, a personification feedback behavior is synchronously performed;

[0020] The humanized feedback behavior includes adjusting towards the user's line of sight, indicating a target object in the image frame through virtual limb movement, or superimposing a target marker on the real-time picture.

[0021] Further, the method dynamically monitors network connection state and image frame quality.

[0022] When detecting that the network delay exceeds a threshold or the image clarity is lower than a standard, the video capture parameters and inference strategy are adaptively adjusted.

[0023] In a second aspect, the present application further provides a real-time two-way visual interaction system of digital human calling camera, which executes the method of the first aspect when running, and comprises:

[0024] A camera capture module for capturing image frames of the user and their environment in real time.

[0025] A question acquisition module for receiving user questions in the form of voice and / or text.

[0026] A key frame screening module for screening semantic key frames from real-time image frames based on the semantic content of the questions.

[0027] A multi-modal question and answer processing module for performing semantic correlation analysis on the questions and image frames and generating answer results.

[0028] A digital human interaction presentation module for driving a virtual digital human to output answer results in voice and synchronized animation.

[0029] Further, the system further comprises a mobile terminal and a cloud server.

[0030] The mobile terminal performs image capture, question acquisition, and key frame screening, and transmits the questions and screened image frames to the cloud server.

[0031] The cloud server performs multi-modal semantic fusion analysis and returns the answer results.

[0032] Further, the multi-modal question and answer processing module comprises a visual encoder and a language encoder for extracting visual features of image frames and semantic features of question texts, respectively, and fusing the visual features and semantic features through a cross-modal attention mechanism.

[0033] The deep combination of visual information and language information is realized by using a multi-modal large model, and higher intelligent level is shown in understanding complex questions and scenes. Compared with the existing system of visual recognition and language answer, the method directly extracts visual evidence and reasons based on the question semantics, avoiding the loss and misunderstanding in the information transmission process. In addition, since the cloud model can integrate massive knowledge, the answer given by the digital person is rich in content and has explanatory, and necessary explanations are also attached, improving the knowledge and interest of the interaction.

[0034] Further, the system further comprises an abnormality monitoring and processing module for monitoring network connection state and image quality in real time, and triggering dynamic optimization of video acquisition strategy and reasoning process when network delay anomaly or image distortion is detected.

[0035] In a third aspect, the present application also provides a computer device comprising a management platform and a memory, the management platform being connected with the memory, the memory being used for storing a computer program, and the management platform being used for executing the computer program stored in the memory, so that the computer device executes the method for realizing real-time two-way visual interaction of digital person by calling camera.

[0036] In a fourth aspect, the present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a management platform to realize the method for realizing real-time two-way visual interaction of digital person by calling camera.

[0037] The present application realizes real-time collection of environment video stream by mobile terminal camera, performs key frame screening by using heuristic scoring function based on the semantic content of user question, and only extracts image frames or features highly related to the question; the screened visual data and user question are transmitted to the cloud, high-level features of the image are extracted by visual encoder, text features are generated by language encoder, and cross-modal attention mechanism is used to realize feature alignment and joint reasoning to generate answer text; the digital person outputs the answer text by voice synthesis and synchronous personification animation and forms an interactive closed loop combined with spatial feedback. The system realizes collaborative optimization of terminal lightweight preprocessing and cloud deep semantic fusion, controls the end-to-end delay in a very low range, and enhances the interactive immersion relying on the multi-modal feedback mechanism, significantly improving the real-time environment perception and natural interaction ability of the digital person in intelligent customer service, AR guide and other scenes.

[0038] Advantages

[0039] By implementing the method and system for realizing real-time two-way visual interaction of digital person by calling camera provided by the present application, the following technical effects are achieved:

[0040] (1) The application greatly shortens the response time from user asking to digital human giving reply through key frame screening on the terminal side and end-cloud collaborative processing. In a typical 4G / 5G network environment, nearly instant feedback can be achieved. Compared with the traditional method of uploading the entire video to the cloud for analysis, this method only transmits and processes less than half of the frame numbers to return the answer. Even if complex visual question answering is performed on a mobile device, the interaction process is almost free of obvious lag or long waiting. The digital human can answer questions immediately, greatly improving the fluency and naturalness of the interaction.

[0041] (2) The device computing and network communication burden is reduced through multi-level optimization. On the one hand, the lightweight local processing of the terminal avoids uploading large amounts of raw video data to the cloud, but only transmits the key information, greatly reducing the data transmission volume. On the other hand, the terminal side only performs lightweight operations, and heavy computing tasks such as deep neural network inference are completed by the cloud, saving local CPU / GPU computing power and power. Since the data sent is more concise, the network bandwidth occupation and user traffic consumption are also smaller, making the system more economical and efficient in a wireless network environment. While meeting real-time requirements, it also achieves a balance between low resource occupation and high performance response, adapting to the deployment requirements of mobile devices or embedded devices.

[0042] (3) The use of cross-modal large models enables deep integration of visual information and language information, with higher intelligence in understanding complex questions and scenes. End-to-end cross-modal fusion enables the system to handle various types of visual question answering. Compared with the existing separate solution of "visual recognition first and then language answer", the application directly extracts visual evidence based on question semantics and jointly reasons, avoiding information loss and misunderstanding in the transmission process. In addition, since the cloud model can integrate massive knowledge, the answers given by the digital human are rich in content and explanatory, and necessary explanations can be added, improving the knowledge and interest of the interaction.

[0043] (4) The application creates a simulated "face-to-face" interaction experience. Users are no longer limited to preset instructions or button operations, but can freely communicate with digital humans through natural language and body movements; digital humans respond in a personified manner, with a much higher degree of freedom and immersion than traditional human-computer interaction methods. In addition, digital humans give emotional temperature to communication through rich facial expressions, body movements, and tone changes. Compared with a pure text question and answer interface, the system can effectively improve user satisfaction, and the average single use time and repeat use rate of users also increase significantly. BRIEF DESCRIPTION OF DRAWINGS

[0044] To make the above-mentioned camera-invoking digital human real-time bidirectional visual interaction method and system of the present application more obvious and easy to understand, the drawings needed in the specific embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can also be obtained from these drawings without creative labor for those skilled in the art.

[0045] Figure 1 is a flowchart of the digital human real-time bidirectional visual interaction method of the present application.

[0046] Figure 2 is a terminal-cloud collaborative architecture diagram of the digital human real-time bidirectional visual interaction system of the present application.

[0047] Figure 3 is a structure diagram of the visual language multi-modal semantic fusion model of the present application. DETAILED DESCRIPTION

[0048] Embodiment 1:

[0049] A camera-invoking digital human real-time bidirectional visual interaction method is provided, and the method flow is as shown in Figure 1 , which includes:

[0050] Step one, the terminal device calls the camera to collect the real-time video stream of the user and the surrounding environment at a high speed. The camera acquisition module dynamically adjusts the frame rate and resolution according to the current network condition and terminal computing resources, and performs basic preprocessing on the obtained original video frames to ensure the stability of subsequent analysis and image quality.

[0051] Step two, when the user raises a question in the form of voice and / or text, the system obtains the question and associates it with the current visual content captured synchronously within the camera field of view. For voice questions, the system first converts the voice into text in real time through automatic speech recognition, thereby obtaining the question text corresponding to the visual content. This step realizes the semantic binding of the user's natural language question and the camera picture, ensuring that the subsequent processing focuses on the visual area of user's attention.

[0052] Step three, the visual perception module on the terminal side intelligently filters and preliminarily identifies real-time video frames to filter out redundant information frames and extract key information frames highly relevant to the question semantics. On the one hand, the module selects a subset of frames that may contain clues to the answer from consecutive frames through heuristic algorithms; on the other hand, necessary lightweight identification is performed on the selected frames. Through this intelligent key frame extraction strategy, the terminal only needs to process and upload a small number of key frames to obtain the visual basis required to answer the question, avoiding uploading the entire video stream to the cloud for processing, thereby greatly reducing the communication bandwidth occupation and terminal computing load. After completing key frame extraction and preliminary identification, the local module sends the selected key frame images or extracted visual features to the network communication module for the next step.

[0053] Step four, the terminal sends the user's question text and the locally filtered key image frames to the cloud-side question and answer processing module through the network communication module. After receiving the data, the cloud uses a visual language multi-modal semantic understanding model to perform cross-modal fusion reasoning on the image and text: on the one hand, the cloud's visual encoder extracts high-level semantic features of the image; on the other hand, the language encoder encodes the question text into a text feature vector. Then the features of the two modalities are mapped into a unified high-dimensional semantic space for deep joint reasoning analysis, and finally the answer text of the question is generated. After the cloud completes the answer reasoning, it immediately sends the result back to the terminal device through the network. Since the terminal has pre-filtered and compressed the amount of data uploaded, the cloud can complete complex reasoning under lower bandwidth occupation by taking advantage of its powerful computing power, and the end-to-cloud collaboration ensures high accuracy and low delay of the question and answer.

[0054] Step five, the digital human interaction presentation module on the terminal receives the answer text returned by the cloud and feeds back the answer to the user in a personalized and multi-modal combined manner, including: using text-to-speech synthesis technology to convert the answer text into natural and fluent speech; driving the virtual image of the digital human to generate lip movements and facial expressions in real time to broadcast the answer; if necessary, combining appropriate body movements or visual aids to enhance the feedback effect. For example, when answering a question about a certain object in the camera view, the digital human makes a gesture pointing to the object, or highlights the relevant area on the real-time screen of the terminal and superimposes text explanations.

[0055] A digital human real-time bidirectional visual interaction system calling a camera is also provided, which includes:

[0056] The camera acquisition module, deployed in the mobile terminal, is used to access the device's camera to acquire real-time video streams or still images. This module supports various camera types and resolution settings, and dynamically adjusts the acquisition frame rate based on network and terminal computing resources to balance image clarity and real-time interactivity. The camera acquisition module performs preliminary image processing on the acquired video frames, then provides the pre-processed frame data to the local vision processing module. This module acts as the visual sensor for a digital human, continuously providing the system with high-quality environmental visual information.

[0057] The visual perception and keyframe filtering module, deployed on the terminal device, performs real-time lightweight analysis and processing on the raw image data captured by the camera, including but not limited to image compression and downsampling, basic target detection and recognition, and intelligent keyframe extraction algorithms. This module introduces a language-semantic-guided keyframe filtering strategy, which can intelligently filter video frames to extract key information based on semantic clues in user queries. Specifically, the local module uses a language-semantic-driven heuristic scoring function to score and sort the frame sequence, prioritizing frames with higher scores, skipping redundant frames, and stopping further frame analysis when it determines that sufficient visual key information has been collected, outputting the required visual feature results. The keyframe scoring function is formally represented as:

[0058]

[0059] In the formula, For the first Keyframe scoring of frame images; This is a heuristic scoring function that calculates a score based on the degree of matching between image features and question semantics. For the first Frame Image Functions for extracting visual features; For user question text The module performs semantic encoding. Through the intelligent frame selection mechanism, it can adaptively select frames that are highly relevant to the question-and-answer semantics, significantly reducing the amount of data that needs to be processed and transmitted subsequently, while ensuring that no key information related to the question is missed. During local visual recognition, the module also calls the AI ​​acceleration hardware of the terminal device to execute a lightweight pre-trained model to speed up processing and enrich the recognition dimensions. It should be noted that the basic visual recognition technology used by this module can adopt existing mature algorithms, and the implementation method is not limited to a specific model. The module's intelligent keyframe extraction and semantically relevant filtering mechanism ensures that the system can correctly understand the semantic focus of the user's question in the current scenario, even if it only processes a very small number of frames.

[0060] A network communication module is responsible for managing data transmission and interaction between the terminal and the cloud server, ensuring that the key information processed locally can be safely, efficiently and low-latency uploaded to the cloud, and the results returned by the cloud are promptly issued to the terminal interface. The communication module is preferably implemented using a low-latency asynchronous request-response mechanism based on HTTP / HTTPS or WebSocket protocol. In terms of data transmission, the module packages the user's question text and selected key frame images together, and performs necessary encryption and compression before uploading to reduce bandwidth occupancy; for larger image data, formats such as JPEG, WebP are used for further compression, or only the high-level semantic features extracted after encoding are sent to save bandwidth. To make full use of the network round-trip time, the communication module works in a non-blocking manner, while waiting for the cloud response, the terminal can perform subsequent preparation in parallel to maximize the use of communication waiting time. This module pays special attention to real-time optimization and data security, has an anti-jitter transmission strategy to minimize the impact of network delay, and takes necessary encryption measures for the data throughout the transmission process to prevent privacy leakage.

[0061] A cloud question and answer processing module is deployed on the cloud server. After receiving the user's question text and related visual data from the terminal, the module first maps the image features and text features to a unified representation space through a multi-modal semantic fusion model, achieving cross-modal understanding and alignment. In a preferred embodiment, the cloud module loads a specially trained visual question and answer model, which contains a visual attention mechanism inside, dynamically focusing on relevant areas in the input image according to the question, thereby accurately capturing the association between image semantics and question intent. Once the multi-modal model fuses the question text and image information to complete semantic analysis, the internal generative reasoning unit generates a natural language text answer. During answer generation, the cloud module also calls the knowledge base or network search as needed to supplement additional information. The entire cloud reasoning process is optimized to reduce response latency while ensuring accuracy, such as using a hierarchical reasoning strategy, first giving a preliminary answer quickly by a smaller model, and then refining the answer by a large model to improve the quality of answers to complex questions; or pruning and quantizing the model to improve reasoning speed. In the multi-round dialogue scenario, the cloud module also supports context association and result caching mechanisms. For the user's continuous related questions, the previous dialogue content is input into the model as context, or the recent processing results are cached to avoid repeated reasoning, thereby improving the coherence and interaction efficiency of the dialogue.

[0062] During the model training phase, the system employs a cross-modal semantic alignment contrastive learning strategy to optimize the fusion representation of visual and textual features, enabling the model to better understand both image and linguistic information simultaneously. For example, it brings correctly matched image-text pairs closer together and separates irrelevant pairs. Specifically, assuming the training batch contains a set of image feature vectors and corresponding text feature vector sets, a symmetric cross-modal contrastive loss function is defined to bring matched image-text pairs closer together and distinguish unmatched pairs:

[0063]

[0064] In the formula, This represents the cross-modal contrast loss value. This represents the number of samples in the training batch. For the first Visual feature vectors of the images; For image Matching the feature vector of the problem text; Image features Text features Cosine similarity between them; Temperature coefficient; This is a counter. By minimizing the loss function, the distance between matched image-text pairs in the semantic space can be reduced, and mismatched image-text pairs can be separated, thereby significantly improving the model's alignment understanding ability across visual and linguistic modalities.

[0065] The digital human interaction presentation module is deployed on the terminal device and is used for presenting the answer returned by the cloud to the user in a personified and multi-modal manner. The module includes a speech synthesis unit, a virtual human rendering unit, and a multi-modal feedback mechanism, etc. The speech synthesis unit converts the answer text into natural and fluent speech by using a text-to-speech technology; the virtual human rendering unit drives a pre-constructed digital human model to generate lip movement and facial expression animation synchronized with the speech, so as to ensure that the voice and the lip movement of the picture are strictly consistent. Preferably, the digital human image makes corresponding facial expressions or body movements according to the content and tone of the answer, for example, smiling when answering a happy question and looking serious when explaining a serious topic; if the answer involves pointing to an object, the digital human makes a gesture of pointing to the object or turns its head to gaze in the corresponding direction. The module is also responsible for providing visual auxiliary feedback. When the question answered by the digital human is closely related to the real-time picture of the camera, the relevant image or text description is displayed on the system interface at the same time, and the picture area of interest of the user is highlighted. For example, when the user asks the camera "What is this flower?" and there is a flower in the picture, the digital human answers "This is a red rose" at the same time, and the real-time video picture on the interface will mark the flower with a box and add the word "rose" to it, realizing the intuitive interaction of text and pictures. This dual feedback design of language and vision enables the user not only to hear the answer but also to see the basis of the answer directly, greatly enhancing the immersion and accuracy of the interactive experience.

[0066] The above modules are designed in a modular manner, clear division of labor and close cooperation, forming a complete end-cloud collaborative system architecture. On the one hand, the modules interact with each other through standardized data interfaces, supporting loose coupling and flexible expansion, and developers can replace or upgrade certain modules without modifying the implementation of other modules. On the other hand, the modules are distributed and deployed in the terminal or the cloud according to performance requirements, forming a real-time interaction system of end-cloud collaboration. The computationally intensive cross-modal inference tasks are executed on the cloud, while the tasks with high real-time requirements such as camera acquisition and rendering feedback are retained on the terminal. The local and cloud are connected through a high-speed network interface, making the system not only take advantage of the computing power of the cloud but also minimize the impact of network delay. Such architecture design effectively integrates the elements of camera visual perception and digital human interaction, providing a solid and flexible technical foundation for the large-scale application of digital human technology.

[0067] The key algorithm mechanism used in the present application includes functions and principles as follows:

[0068] Image sharpness scoring function:

[0069] The image sharpness is measured by calculating the gradient amplitude energy of the image. Let denote the intensity value of the gray image at pixel , then the sharpness score of the frame is The global average gradient energy can be defined as:

[0070]

[0071] wherein, is the image sharpness score; and are the height and width pixel number of the image, respectively; and are the gradient values of the image in the horizontal and vertical directions at The formula actually calculates the average value of the edge gradient strength in the entire image: the clearer the image, the more distinct the internal edge contour, the greater the gradient amplitude, and the higher the score obtained; conversely, if the image is blurred, the gradient value is generally small, and the score will also be reduced. The system is provided with a sharpness threshold value, when it is detected that the sharpness of a frame is lower than the threshold value, it is considered that the quality of the frame is poor, and the local module discards it and does not send it to the cloud as a key frame; if multiple frames are blurred in succession, the user is prompted to adjust the camera focal length or increase the ambient light to ensure that the frame image used for analysis is clear and available.

[0072] Image language joint attention weight calculation:

[0073] In the cloud multimodal model, a cross-modal attention mechanism is introduced to measure the correlation between the image and the text. The system uses an attention model that interacts between the visual Transformer and the text Transformer. Assuming that the visual encoder extracts feature vectors of the image , and the text encoder generates hidden feature representations of the question sentence , then for the th text feature, the attention weight of the th region feature of the image is defined as:

[0074]

[0075] wherein, is the attention weight of the th text feature to the th image region; is the semantic feature vector of the th word / position in the question text; is the visual feature vector of the th image; is the total number of regions into which the image is divided; is an exponential function used to normalize the weight. The above weight constitutes a normalized probability distribution. Through calculation, the ​, reflecting the distribution of attention of each part of the question text on different image regions. Based on these attention weights, the model locates the region in the image that is most relevant to the semantics of the question, thereby improving the accuracy of cross-modal understanding. For example, the matching degree score of the entire image and the entire question text is further calculated: the attention weights of each position of the text on the image are applied to the corresponding image region features to obtain a fused representation, and then the average value of the similarity between the text overall features is calculated, i.e. a scalar value that measures the semantic matching degree of the image content and the given question. The higher the value, the more semantically consistent the image content is with the given question. It should be noted that the above attention calculation is usually completed once in the form of matrix operation inside the model, but its effect is equivalent to calculating and normalizing the weights element by element according to the above formula. Using this cross-modal attention mechanism, the model can dynamically focus on the most relevant region in the image to the question, thereby more accurately capturing the association between visual semantics and question intent.

[0076] Model output confidence evaluation mechanism:

[0077] To evaluate the credibility of the answers given by the cloud question and answer model, a confidence score method for the output results of the model is introduced. In general, the confidence is determined by the probability distribution of the model when generating the answer. For example, if visual question and answer is regarded as a classification problem with candidate answers, the model outputs a score for each candidate, and the confidence of the model for the final predicted answer is defined as the Softmax probability corresponding to the selected option:

[0078]

[0079] wherein is the confidence of the model for the predicted answer; is the logit value of the predicted answer; is the total number of candidate answers; is the logit value of the th candidate answer.

[0080] If the model outputs indefinite-length free text answers in a generative manner, the confidence is also calculated in a similar manner. For example, the entire generated answer is regarded as a joint event of a series of words, and the probability of the model outputting the sequence is used as the confidence score of the answer. The system sets a confidence threshold, and when the confidence of the model for a certain answer is lower than the threshold, a preset backup strategy is triggered to avoid giving misleading answers when the model is not confident enough. Through the above confidence evaluation mechanism, the system obtains a reliability index while generating the answer, which is used for subsequent module decision-making on how to present the answer. When the confidence is high, the answer is directly given normally, when the confidence is low, the tone and expression of the digital person are adjusted appropriately, or an uncertain prompt is given, thereby improving the reliability and user trust of the interaction.

[0081] A schematic diagram of the edge-cloud collaborative architecture is shown below. Figure 2 As shown, the data flow and time sequence of each module working together under normal circumstances are as follows:

[0082] After a user launches the digital human application on their mobile device, the camera begins continuously capturing video of the surrounding environment. A typical interaction scenario involves the user pointing the camera at a scene or object and asking the digital human a question via voice or text. Only when the system detects a question does the local visual perception module perform the aforementioned keyframe extraction and preliminary recognition on the current video frame sequence, extracting the image content related to the question in real time. For example, if a user asks, "What kind of flower is this?", the system immediately captures the frame at the time of the question and quickly identifies the main flower as the focus locally on the device. Similarly, if a user asks, "What is he doing?", the local module selects several representative frames from the continuous video, capturing key moments of the person's actions to understand their behavior. Subsequently, the network communication module packages the user's question text and the selected keyframe data and sends them to the cloud.

[0083] Visual language multimodal model architecture, such as Figure 3 As shown, after receiving a request, the cloud-based question-answering module first extracts visual features from the image using a visual encoder, and then feeds these features, along with the text question, into a multimodal model for joint encoding and reasoning. Internally, the model uses mechanisms such as self-attention and cross-modal attention to understand the relationship between the image and text information, thereby generating the answer statement. During the generation process, the model will automatically call its knowledge base or online search if needed to identify the type of flower or understand the meaning of a person's actions. These complex reasoning processes are transparent to the end user. Once the cloud-based system generates the answer text, it immediately sends the result back to the terminal device via the network.

[0084] After receiving the answer text, the digital human interaction module on the terminal immediately calls the text-to-speech engine to synthesize the corresponding speech and drives the virtual digital human's facial and motion models to ensure that its lip movements are strictly synchronized with the speech, broadcasting the answer to the user in real time. For example, when the digital human answers "This is a red rose," the virtual human image in the application interface will turn its head to look at the location of the flower in the camera's view and point to it; or it will directly select the location of the flower on the live camera view and label it with its name. The entire process, from the user asking a question to the digital human completing the answer, usually takes only a few seconds or even less than one second, achieving a near real-time interactive experience.

[0085] The application realizes high-performance real-time visual question answering through seamless collaboration between the terminal and the cloud. The terminal fully filters and refines the information before uploading, reducing the computational load and communication bandwidth pressure of the cloud. The cloud leverages strong deep learning inference capabilities to solve the core problem of visual semantic understanding. It is this architectural design that enables complex visual question answering applications to run smoothly on mobile devices with limited computing and battery resources, while maintaining a smooth and natural user experience. The method and system effectively overcome the difficulties of real-time interaction between vision and language in the prior art, have high real-time performance, high accuracy, and good user immersion experience, and can be widely applied to intelligent customer service, virtual assistants, remote education, and other scenarios, with significant practical value and promotion prospects.

[0086] Given the need for digital human applications to be widely deployed on mobile devices, computer web pages, and other platforms, a unified adaptation scheme and verification mechanism are developed for each platform to ensure consistent system functionality and stable performance across platforms. Main measures include:

[0087] At the architectural design level, each sub-module follows platform-independent interface specifications to encapsulate and shield the underlying differences between different systems. For example, the camera capture module internally calls different native APIs for different platforms, but provides a unified frame data format and frame rate control interface to the upper layer, ensuring consistent behavior of the camera capture function on different platforms. For another example, the virtual human rendering module is implemented based on a cross-platform graphics rendering engine. Mobile devices use Metal or OpenGL ES interfaces, while web devices use WebGL or WebAssembly technologies. Good encapsulation ensures that the rendering and interaction control interfaces of the digital human image remain consistent. Through these module encapsulations, developers do not need to worry about platform details, and each module can run equivalently on different systems.

[0088] The terminal-cloud collaboration architecture naturally supports devices of different performance levels by dynamically adjusting the task division between the terminal and the cloud, achieving approximately consistent response speed across platforms. On high-performance terminals, more real-time recognition tasks are processed using local computing power. In low-performance terminals or web browser environments, more reliance is placed on cloud computing power, with the terminal only performing light work. The system automatically adjusts the depth of processing on the terminal side based on device performance and current network conditions. For example, when network bandwidth is limited or terminal performance is insufficient, the frame rate or resolution of the camera video capture is reduced, and only the necessary features are extracted for uploading to reduce data transmission volume and computing delay. Conversely, when the network is good and the terminal performance allows, the video quality is improved or the local processing volume is increased to obtain more rich semantic information. This adaptive strategy ensures that the response time of the system remains within an acceptable range regardless of the device used by the user in various network environments, achieving balanced and stable real-time interaction experience across platforms.

[0089] In terms of user interface and interaction logic, a unified design specification and adaptation scheme is adopted to ensure consistency in the layout of digital human interface, virtual human behavior logic, and user interaction methods across platforms. During development, a cross-platform testing framework is used to test Apple applications, Android applications, and web applications simultaneously. The display effects and response delays on different operating systems and devices are compared to ensure that there are no missing functions or inconsistent experiences. If special compatibility issues are found on a platform, a special adaptation branch is added to the platform's module implementation for optimization, while the main process logic remains unchanged. In summary, whether users use Apple, Android mobile devices, or computer browsers, they can obtain a digital human visual interaction experience that is functionally equivalent, smooth in performance, and unified in interface.

[0090] To address various abnormal situations that may occur during real-time two-way visual interaction, a comprehensive abnormality detection and processing mechanism is designed to cover network communication, image acquisition quality, cloud reasoning process, and terminal running state, ensuring the system's stable operation in complex environments. This includes:

[0091] During communication between the terminal and the cloud, if unstable network connection or high round-trip delay is detected, the communication module will activate fault-tolerant strategies. First, a timeout retransmission mechanism is used, which automatically resends requests or key data frames when no response is received from the cloud within a predetermined time after sending the request, ensuring that requests can be resent in the event of occasional packet loss. Second, for persistent high latency or disconnection, the system supports rapid network channel switching or service degradation strategies. For example, it automatically switches from WiFi to cellular mobile networks, or reduces the frequency and resolution of video frame transmission to alleviate network pressure when bandwidth is insufficient. If the network is interrupted for too long and cannot be restored in time, the system will promptly inform the user of the current network anomaly and temporarily enter offline waiting mode, automatically continuing the unfinished question and answer process after connection is restored. Through the above measures, even in the case of network fluctuations, the system can ensure the completion of the question and answer task or at least provide a prompt when problems occur rather than crashing directly.

[0092] To address the quality issues that may occur in the image frames captured by the camera, a quality detection and adaptive adjustment mechanism is integrated in the local vision processing module. On the one hand, the system calculates indicators such as sharpness and brightness for each frame of image and compares them with pre-set thresholds: if it finds that the current frame has excessively high blur or severely insufficient brightness, it determines that its reliability is low and does not use it as a key frame to send to the cloud; when multiple consecutive frames do not meet the standards, the terminal interface will prompt the user to adjust the camera or the shooting environment. On the other hand, the camera acquisition module automatically adjusts the camera parameters to improve the image quality when it detects that the frame quality is not good, such as enabling automatic focusing to refocus, enabling automatic exposure compensation to improve brightness, etc. If some frame data is severely distorted or damaged, the system will discard the abnormal frame and wait for the next frame of data, avoiding passing abnormal data to the subsequent module for analysis. Through these strategies, the quality of the image frames entering the question and answer analysis process is ensured to meet the standards, thereby improving the accuracy of visual understanding.

[0093] In the cloud question and answer processing module, to address the situation where large multi-modal models may have response timeouts or generate unreliable results, a timeout monitoring and result verification mechanism is set. The cloud starts timing when it begins processing the request, and if the inference time exceeds the pre-set threshold, it is determined to be a possible timeout. At this time, the system has two coping strategies: one is to abort the inference process of the current large model and use a backup lightweight model or a pre-set answer to reply, for example, return a polite prompt that cannot answer to ensure timely response to the user; the second is to continue to wait for the large model to complete the inference, but at the same time send progress feedback to the user interface, and present the answer immediately after the result is generated. In addition to timeout monitoring, the cloud module also performs effectiveness verification on the answer results returned by the model, such as detecting whether the answer is empty or obviously inconsistent with the question semantics. If the result is found to be invalid, the system will record the exception and try to call the model to generate again, or enable a different inference path to solve again; if multiple retries still fail, the cloud eventually returns an error flag, and the terminal presents the error information or guides the user to retry the question later in a friendly manner. Through the above timeout and result verification processing, the long-time non-response or answering the wrong question caused by model abnormalities is avoided, ensuring the robustness of the interaction process.

[0094] On the terminal side, the small AI model relied on by the local visual processing module may fail to load, initialize incorrectly, or crash during operation, and the corresponding remedial strategies are designed. Once it is detected that the terminal AI model cannot be normally loaded or called, the system will immediately switch the working mode, and try to downgrade as much as possible but not lose the core functions. The first measure is to bypass the faulty module and take a simplified process, for example, to skip the local text recognition step and directly send the image frame to the cloud for text recognition, so as to ensure that the question and answer main process is not interrupted. If the function of the faulty module is indispensable to the whole, the standby service of the cloud is called to replace the work of the terminal model, although this may increase some delay, but can maintain the functional integrity. For example, when the terminal face recognition model is unavailable, the key frame can be uploaded to the cloud to request the cloud to assist in recognition. At the same time, the terminal will record error logs and notify the application management module for subsequent repair. If the terminal AI inference result is detected to be abnormal during operation, the system will also take isolation and degradation strategies: if necessary, temporarily disable the complex analysis of the terminal, only keep the core functions of the camera frame forwarding and the cloud question and answer, and remind the user that the terminal performance is insufficient or the corresponding function is temporarily unavailable. Through the above measures, the invention minimizes the impact of terminal AI module abnormalities on the system, and ensures that even if some local intelligent functions fail, the main question and answer interaction of the digital person can still run basically, so as to not interrupt the user experience.

[0095] In summary, through the comprehensive abnormal handling mechanism design, the invention covers potential abnormal conditions of each link from network communication, image quality to model inference, and device state, and provides comprehensive and effective measures to deal with it, so as to ensure that the system still runs stably and reliably under various complex conditions, and provides users with a sustained good interactive experience.

[0096] Embodiment 2:

[0097] This embodiment shows the application of the system in a real-time video conversation scene. The user starts a digital person application with a personified virtual image on a mobile terminal, and interacts with it like a video call. The front camera of the terminal continuously captures the user's facial expressions and the scene of the environment, and provides it to the digital person system; the user asks the digital person questions through voice. The digital person "hears" the user's question, and "sees" the scene where the user is through the camera, and gives an answer that fits the scene.

[0098] During the specific interaction, the camera acquisition module continuously acquires picture frames of the user and the environment. The local visual processing module analyzes these frames in real time to recognize the user's facial expressions, body movements, and objects in the background scene. When the user asks a question, the system first transcribes the speech into text through automatic speech recognition, then performs video frame filtering locally on the terminal, for example, selects a frame from the most recent pictures in which the portrait on the wall behind the user is clearest, and detects the face of the person in the portrait on this frame. Then the terminal sends the processed question text and the frame image to the cloud. After receiving them, the cloud multi-modal model identifies who the person in the portrait is through the face recognition sub-model, and understands that the user is asking the identity of the person in the portrait based on the question semantics. If the portrait is identified as a historical figure, the model retrieves the information of the figure from its own knowledge base or online encyclopedias, and finally generates an answer text. After the answer is transmitted back to the terminal, the digital human interface feeds back the result to the user in a personified way, by voice broadcasting the above content, while the virtual human image turns its head to the direction of the portrait behind the user and makes an indicating action. The user thus feels that the digital human has indeed "seen" the painting and correctly identified the person in the painting, realizing a highly natural situational question and answer interaction between the user and the digital human. Through the camera visual input, the digital human can recognize the entity environment around the user and understand its relevance to the user's question; the collaborative processing of the terminal and the cloud ensures the real-time identification and answering; the digital human feeds back the answer through natural expressions and body movements, enhancing the user's trust in its accuracy and willingness to continue the communication.

[0099] Embodiment 3:

[0100] This embodiment introduces the application of the system in daily life scenarios, i.e., the digital human as a personal intelligent assistant. The user can have real-time visual communication with the digital human assistant on terminals such as mobile phones, tablets, etc. Typical application scenarios include:

[0101] When the user is traveling, he / she opens the camera to take pictures of the surrounding scenery and asks questions. The digital human assistant can identify the landmark buildings or historical sites in the taken pictures and explain the relevant historical background and information in real time. For example, the user asks "Who is this?" pointing at a statue, and the digital human will identify the person represented by the statue and then answer his / her name and life story, accompanied by a virtual gesture pointing to the statue.

[0102] When the user is shopping, he / she scans the goods with the camera and asks the digital human about the information of the goods or whether there are more favorable prices. The digital human reads the text and signs on the packaging of the goods through text recognition technology, identifies the brand and model of the goods, and instantly queries the price comparison information online, then tells the user the online price of the goods or recommends alternative products. Throughout the process, the digital human obtains the bar code and packaging information of the goods through visual acquisition, realizes intelligent analysis of the seen goods, and provides personalized shopping suggestions.

[0103] When users encounter device connection or usage problems, they can simply point the camera at the relevant device and describe the symptoms to the digital person. The digital person observes the visual information such as the indicator light state, interface connection, etc. of the device panel, and gives fault diagnosis and operation guidance in combination with the user's language description. For example, when the computer cannot be connected to the network, the user uses the camera to shoot the status indicator light of the router, and the digital person prompts the possible cause after recognizing the light flashing mode. Through the understanding of the environment by vision, the digital person assistant greatly improves the efficiency and ease of use of remote technical support.

[0104] Through the above scene applications, the digital person assistant can understand the real situation in which the user is located, and provide a what-you-see-is-what-you-get type of service in combination with voice dialogue, becoming a considerate and intelligent consultant around the user.

[0105] Embodiment 4:

[0106] This embodiment applies the system in a home environment, acting as a smart home housekeeper. A network of cameras is deployed in the home, and the digital person housekeeper monitors the home situation in real time through the cameras and communicates with the occupants, specifically including:

[0107] When an abnormal situation occurs in the home, the digital person housekeeper will automatically call the relevant camera to view the scene. If the camera screen shows only pet activity, the digital person will notify the owner through the loudspeaker voice that there is no need to worry; if a suspicious intruder is detected, the digital person will immediately issue a warning and trigger the alarm system, while recording the evidence video through the camera. In the case of an old person staying alone at home, if an accidental fall of the old person is detected, the digital person can discover through vision and immediately issue a voice inquiry, and automatically dial an emergency call when there is no response. With the help of visual empowerment, the digital person housekeeper can vigilantly guard the safety of the home 24 hours a day.

[0108] Family members can directly dialogue with the digital person housekeeper through the central control device with a screen. For example, saying to the digital person on the screen "what's in the fridge?", while sharing the camera screen inside the fridge to the digital person. The digital person checks the food in the screen and answers the inventory in the fridge, and further proposes a recipe suggestion. For example, when the owner goes out and asks "are the windows closed?", the digital person will call up the camera in each room to confirm the window status one by one, and through the voice in combination with the highlight of the corresponding camera screen on the tablet interface, informs the owner of the closing status of each window, avoiding safety hazards.

[0109] The digital person can also recognize facial expressions and body language through the camera, and pay attention to the emotional health of family members. For example, when detecting that an old person living alone is unusually silent or has a depressed expression, the digital person will actively appear through the screen in the home, chat and comfort him, or remind his family members to pay more attention. When a child comes home from school and detects that the user is in a high-spirited mood, the digital person will smile and interact, like a family member, giving a warm response and care.

[0110] The application of the system in the smart home scenario enables the intelligent assistant to not only understand the instructions of the occupants, but also understand the environmental changes and the state of the users, thereby providing more personalized and safe and reliable home services.

[0111] Embodiment 5:

[0112] This embodiment applies the system to the guide tour and commercial guide service in public places:

[0113] In museums, art galleries and other venues, a digital human guide is displayed on a large screen. Cameras arranged in the venue capture the positions and behaviors of the audience in real time. When an audience member approaches a certain exhibit and stops to watch, the digital human guide will "walk" towards the direction of the exhibit that the audience member is watching and begin to explain the background story of the exhibit. If the camera captures signs that the audience member is puzzled or has a question, the digital human will adaptively adjust the depth of explanation and the speaking speed to provide targeted explanations until the audience member shows an understanding expression or nods to indicate that he or she understands. Throughout the process, the digital human understands the audience's focus and reaction through vision, greatly improving the visiting experience.

[0114] In retail stores or exhibitions, the digital human provides services as a virtual guide. Cameras in the store identify the number of customers entering the store, the areas where they stay, and the goods they are interested in in real time. When a customer looks at a certain good for a long time, the digital human guide will greet the customer and introduce the features of the good. If the camera sees that the customer repeatedly gestures to check the size of a piece of clothing, the digital human will actively suggest trying on the clothes or provide inventory inquiries. Thanks to its multilingual ability, the digital human guide can serve customers from different countries without barriers. With visual interaction, the guide service becomes more proactive, considerate and personalized, thereby improving the commercial conversion rate.

[0115] In the above scenarios, the system enables the virtual guide and the virtual guide to have environmental perception ability, which can actively perceive user needs and provide targeted services. Such digital human service personnel will greatly improve the intelligent service level and customer satisfaction of venues and stores.

[0116] Embodiment 6:

[0117] This embodiment applies the system to the education field to empower digital human teachers or coaches to realize two-way interaction in remote teaching and training:

[0118] In an online classroom scenario, a digital human teacher appears as a virtual avatar on the terminal screen of each student to deliver lectures in real time. Through the student's camera, the system allows the digital human teacher to observe the listening state of each student in real time. For example, if the digital human observes that a student frequently frowns, it may judge that the student is not keeping up with the pace, and will temporarily slow down the pace of teaching or ask questions to ensure that the student understands; When a student raises his hand, the digital human will turn his gaze to him and call his name, and then patiently answer the question. If it is found that most students look confused, the digital human will change the teaching strategy and add more examples to help understanding. After class, students can also show their homework to the digital human tutor through the camera, and the digital human automatically identifies the content of the homework for correction, and points out the mistakes through voice explanation and screen annotation. For schools in remote areas that lack teachers, a digital human teacher can be deployed to teach students in multiple places through a network large screen, and combined with camera interaction to achieve individualized teaching. In vocational skill training, a digital human coach can observe the actual operation method of the trainee through the camera, point out the non-standard places in real time and demonstrate the correct actions, and provide personalized guidance and correction.

[0119] The digital human can also serve as a rehabilitation training guide or psychological counseling assistant. Through the camera video, the digital human rehabilitation coach supervises the patient to complete the prescribed rehabilitation movements at home, identifies whether the body posture is in place, and if it finds deviations, it will immediately correct it through voice and demonstrate the correct posture with a virtual image, ensuring the quality of rehabilitation training and preventing secondary injuries caused by incorrect movements. For psychological counseling scenarios, the digital human assistant observes the patient's facial expressions and body language through the camera, such as detecting that the other party looks depressed or cries, and will comfort and encourage softly; and dynamically adjusts the pace and content of the conversation according to the patient's feedback, making the patient feel understood and cared for. Since the digital human can provide 24-hour care and companionship, it helps to improve the continuity of psychological treatment and the patient's sense of security.

[0120] By introducing visual interaction, the digital human achieves comprehensive perception of the learner's state in the field of education and training, and can pay attention to the reaction of each student like a real teacher and guide according to the situation, achieving more efficient and intuitive knowledge transfer and personalized tutoring effect. This ability is particularly valuable in scenarios where there is a shortage of teachers or remote teaching.

Claims

1. A method for real-time two-way visual interaction of digital humans, characterized in that, include: Real-time video streams of the user and their environment are acquired via the camera on the terminal device. Receive natural language questions from users in the form of voice and / or text; Based on the semantic content of the question, the AI ​​acceleration hardware of the terminal device is invoked to run a lightweight pre-trained model to perform keyframe filtering on the image frame sequence of the acquired video stream, filtering out keyframes that are highly related to the semantics of the question, and establishing a semantic association between the question and the real-time image frames; wherein, the keyframe filtering adopts a heuristic scoring function to calculate a score based on the degree of matching between image features and the semantics of the question. The cloud server performs visual-language multimodal semantic fusion analysis based on the question and keyframes, and generates the corresponding answer result; In the semantic fusion analysis process, visual features of the keyframes are extracted by a visual encoder, semantic features of the question text are extracted by a language encoder, and a cross-modal attention mechanism is used to fuse the visual and semantic features to align the question semantics with the image content. The visual encoder and language encoder are trained using a cross-modal contrastive loss function to narrow the distance between matching images and text in the semantic space and separate mismatched image-text pairs. The terminal device outputs the answer result through a virtual digital human using speech synthesis and synchronized animated facial expressions; when the virtual digital human outputs the answer result, it simultaneously performs anthropomorphic feedback behavior, which includes adjusting its gaze toward the user and indicating target objects in the image frame through virtual body movements.

2. The method according to claim 1, characterized in that: The multimodal semantic fusion analysis is achieved through collaboration between terminal devices and cloud servers; The terminal device performs preprocessing and feature extraction on the keyframes, and transmits the extracted visual features and question text to the cloud server; the cloud server performs deep cross-modal reasoning based on the received visual features and question text to generate an answer.

3. The method according to claim 1, characterized in that: The method dynamically monitors the network connection status and image frame quality of the terminal. When network latency exceeds a preset threshold or image clarity is below standard, the video acquisition parameters and question-answering inference strategy are adaptively adjusted to ensure the real-time performance and stability of the interaction. Image clarity is measured by calculating the gradient magnitude energy of the image.

4. A real-time two-way visual interaction system for digital humans, characterized in that: The system executes the method according to any one of claims 1-3 during runtime: The system includes: The camera acquisition module is used to capture video streams of the user and their environment in real time. The question acquisition module is used to receive user questions in voice and / or text format; The keyframe filtering module is used to filter semantic keyframes from the image frames based on the semantic content of the user's question; A multimodal question-answering processing module is used to perform semantic fusion analysis on the question and the selected keyframes, and generate an answer result; The digital human interaction presentation module is used to drive the virtual digital human to output the answer result through speech synthesis and synchronized animation.

5. The system according to claim 4, characterized in that: The camera acquisition module dynamically adjusts the acquisition frame rate and resolution based on the current network conditions and terminal computing resources.

6. The system according to claim 4, characterized in that: The system also includes mobile terminals and cloud servers; The mobile terminal includes the camera acquisition module, the question acquisition module, and the keyframe filtering module, and is used to transmit the question text and the filtered image frames to the cloud server. The cloud server includes the multimodal question-answering processing module, and is used to perform cross-modal semantic fusion analysis based on the received data to generate answer results and return them to the mobile terminal.

7. The system according to claim 4, characterized in that: The multimodal question-answering processing module includes a visual encoder and a language encoder, which are used to extract the visual features of the keyframe and the semantic features of the question text, respectively. The multimodal question-answering processing module also includes a cross-modal reasoning unit, which is used to fuse the visual features and semantic features based on a cross-modal attention mechanism to generate the answer result.

8. The system according to claim 4, characterized in that: The system also includes an anomaly monitoring and processing module, which is used to monitor the network connection status and image frame quality in real time. When an abnormal network latency or image clarity is detected to be lower than a predetermined threshold, dynamic optimization of video acquisition parameters and question-and-answer reasoning process is triggered.

Images