Interaction system in human-machine-virtual human mode

Abstract

The invention belongs to the technical field of virtual interaction, and relates to an interaction system in a human-machine-virtual human mode, which comprises a data acquisition module used for acquiring multi-modal data; the multi-modal data comprises voice data, text data, image data or video stream data; the intention recognition module is used for performing intention recognition on the multi-modal data to obtain an intention recognition result; the intention recognition comprises secret language recognition, question and answer library recognition and large language model recognition; and the virtual human module is used for identifying the identity of the user before acquiring the multi-modal data, inputting the identity information as a cue word into a large language model to adjust an interaction mode, and rendering an intention identification result in a target format in a text streaming manner on a virtual human page. According to the method, the virtual human system driven by the large language model is used as a core medium to interact with a full-stack platform and multiple devices, and rapid and natural operation experience and the capability of efficiently processing point tasks are provided.

Description

Technical Field

[0001] This invention relates to the field of virtual interaction technology, and in particular to an interaction system in a human-machine-virtual human mode. Background Technology

[0002] With the rapid development of information technology, human-computer interaction (HCI) technology has gradually evolved from the traditional single input mode based on keyboard and mouse to more natural, efficient, and intelligent interaction methods. In recent years, multimodal interaction technologies such as voice interaction, gesture recognition, and image recognition have emerged, bringing new possibilities to HCI. However, existing interaction systems still have many limitations when handling multi-device collaborative operation and multi-task processing in complex scenarios.

[0003] Challenges of Multimodal Data Processing: While existing multimodal interaction systems can handle various input data such as voice, text, and images, they still face challenges in data fusion and intent recognition. For example, issues such as the accuracy and real-time performance of voice recognition, the adaptability of image recognition to complex scenes, and the semantic relationships between multimodal data limit the system's intelligence level and user experience. Furthermore, existing systems often require multiple user interactions when handling complex tasks, lacking efficient intent recognition and task execution mechanisms.

[0004] Limitations of Virtual Human Technology: Virtual human technology uses computer graphics and artificial intelligence to create virtual characters with human appearance and interactive capabilities, providing users with a more vivid interactive experience. However, most existing virtual human systems are limited to single-device operation, making it difficult to achieve collaborative interaction between multiple devices or remote devices. Furthermore, when handling complex tasks, virtual humans lack efficient collaborative mechanisms with external systems, data, and tools, thus limiting their application scenarios.

[0005] Insufficiency in Intent Recognition and Task Processing: Existing interactive systems primarily rely on simple keyword matching or predefined question-and-answer libraries for intent recognition, lacking the ability to understand complex semantics and context. Furthermore, in task processing, existing systems often require users to explicitly specify operation steps, lacking the ability to automatically invoke backend tools and execute complex tasks. This forces users to perform a significant amount of manual operation during use, reducing interaction efficiency. Summary of the Invention

[0006] To address the problems existing in the prior art, the purpose of this invention is to provide an interactive system in a human-machine-virtual-human mode, applicable to the scenario of "command and multi-device collaborative work". It utilizes a virtual human system driven by a large language model as the core medium to interact with complex application, model, algorithm, data, cloud and other platforms (hereinafter referred to as full-stack platforms) and multiple devices, providing a fast and natural operating experience and the ability to efficiently handle point-like tasks.

[0007] To achieve the above objectives, the present invention provides the following solution:

[0008] An interactive system in a human-machine-virtual human mode includes:

[0009] The data acquisition module is used to acquire multimodal data, including: voice data, text data, image data, or video stream data.

[0010] An intent recognition module is used to perform intent recognition on the multimodal data and obtain intent recognition results; the intent recognition includes: code word recognition, question-and-answer database recognition, and large language model recognition;

[0011] The virtual human module is used to identify the user's identity before acquiring the multimodal data, input the identity information as prompt words into the large language model to adjust the interaction mode, and stream the intent recognition results as text in the target format on the virtual human page.

[0012] Optionally, the data acquisition module includes:

[0013] The speech recognition submodule is used to acquire the audio stream of the user device and determine the end of the user's speech input by using a preset volume threshold and silence duration. After the user's speech input ends, the audio stream is subjected to audio feature extraction and acoustic model decoding to obtain text information, i.e., speech recognition result.

[0014] The hot word replacement submodule is used to replace the speech recognition results with a preset hot word list to obtain target text information; the hot word list includes proper nouns and special words.

[0015] Optionally, the intent recognition module includes:

[0016] The first intent recognition submodule is used to perform code word recognition on the target text information. When the user intent is determined by recognizing a preset code word or triggering keyword, the task processing link is matched, and the system and application registered on the full-stack platform are automatically called. When the code word recognition fails, the target text information is subjected to question-and-answer database recognition. The target text information is converted into a vector and compared with the question vector of the question-and-answer database. The compared question vector is output, the task processing link is matched, and the system and application registered on the full-stack platform are automatically called. When the semantic similarity does not exceed the preset similarity threshold, the target text information is subjected to large language model recognition. The target text information is input into the large language model. When it is detected that a backend tool needs to be called, the structured data of the function call request is output, the function is executed, and the execution result is returned to the model or sent back to the user in the form of a response message.

[0017] The response message is displayed through the virtual human's UI interface, showing text information, character motion rendering, and voice broadcast, and is used to reply to the user.

[0018] Optionally, the intent recognition module further includes: outputting the compared question vector, including:

[0019] The target text information is converted into a query vector. The most matching semantic tag is retrieved from the vector database of the question-and-answer library according to the vector similarity. This maps the system and application registered by the full-stack platform. At the same time, the vector similarity is used as a matching score, and a candidate list is output in reverse order of matching score.

[0020] Optionally, the intent recognition module further includes:

[0021] The second intent recognition submodule is used to perform a similarity search using knowledge vectors in the vector database when the large language model fails to detect the backend tool. This search retrieves the text fragment most relevant to the independent fragment, replaces placeholders with the most relevant text fragment and the independent fragment to generate a prompt word, inputs the prompt word into the large language model, detects the backend tool to be called, outputs the structured data of the function call request, executes the function, and returns the execution result to the model or back to the user in the form of a response message.

[0022] The vector database slices the original text information to obtain several independent segments. Each independent segment retains overlap information according to a preset overlap field, and the overlapped information is stored.

[0023] Optionally, the virtual human module includes:

[0024] The face recognition submodule is used to acquire the video stream from the camera, extract facial images from the video stream, obtain facial feature vectors, compare the facial feature vectors with the facial features of registered users stored in the face database, calculate the similarity between the facial feature vectors and the facial features, and when the similarity of the comparison results exceeds the facial feature threshold, the face recognition is successful, and a list of user identity IDs that match the facial features is returned.

[0025] The interaction adjustment submodule is used to merge relevant identity information from the user identity ID list into speech, provide feedback information to the user, and input the identity information as prompt words into the large language model to adjust the interaction method.

[0026] Optionally, the virtual human module further includes:

[0027] The virtual rendering submodule is used to stream the intent recognition results as text in the target format onto the virtual human page.

[0028] The speech synthesis submodule is used to convert the text of the intent recognition result into speech and obtain the speech synthesis result.

[0029] Optionally, the speech synthesis submodule includes:

[0030] The speech synthesis unit is used to convert the text of the intent recognition result into speech, obtain the speech synthesis result, correct the pronunciation of polyphonic words in the speech synthesis result, and segment it according to punctuation marks and length. After segmentation, the speech synthesis text is broadcast in a queue.

[0031] The voice interruption unit is used to pause or stop the current voice output and start playing new voice-synthesized content if the user issues a new command during the playback of the voice synthesis result.

[0032] Optionally, the system also includes:

[0033] The remote machine control module is used to encapsulate client operations as API services, package them as standalone programs, install them on a remote machine, display the full-stack platform on the remote machine, and control the remote machine.

[0034] The beneficial effects of this invention are as follows:

[0035] This invention combines the powerful digital intelligence capabilities of a full-stack platform to achieve an innovative intelligent human-computer interaction mode, which can efficiently realize collaborative operation of multiple devices and has significant flexibility and scalability in the process of multi-task execution.

[0036] This invention enables the automated and intelligent processing of complex tasks through intelligent interaction and control of virtual humans, under multimodal input (including voice, gestures, images, etc.), thereby improving the user's interactive experience with the system and devices.

[0037] In collaborative work involving multiple devices and tasks, this invention demonstrates significant advantages and has broad application and promotion prospects in various fields such as smart office, industrial automation, smart home, education and training, and healthcare. Its efficient task execution capabilities, strong scalability, and high degree of user customization give it significant market value and commercial potential in intelligent application scenarios. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of an interactive system in a human-machine-virtual human mode according to an embodiment of the present invention;

[0040] Figure 2 This is a diagram of the virtual human system architecture according to an embodiment of the present invention;

[0041] Figure 3 This is a deployment topology diagram of the virtual human system according to an embodiment of the present invention;

[0042] Figure 4 This is a virtual human interface according to an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of a question-and-answer database according to an embodiment of the present invention;

[0044] Figure 6 This is a schematic diagram of the prompt library according to an embodiment of the present invention;

[0045] Figure 7 This is the main interface of the voice library in this embodiment of the invention;

[0046] Figure 8 This is the voice configuration interface according to an embodiment of the present invention;

[0047] Figure 9 This is the voice recognition-command configuration interface according to an embodiment of the present invention;

[0048] Figure 10 This is a schematic diagram of the image library according to an embodiment of the present invention;

[0049] Figure 11This is a schematic diagram of the perception library according to an embodiment of the present invention;

[0050] Figure 12 This is the main interface of the knowledge base in this embodiment of the invention;

[0051] Figure 13 This is a schematic diagram of knowledge base-document management according to an embodiment of the present invention;

[0052] Figure 14 This is a schematic diagram of the model set in an embodiment of the present invention;

[0053] Figure 15 This is a flowchart of the virtual human voice recognition system according to an embodiment of the present invention;

[0054] Figure 16 This is a flowchart illustrating the human-computer interaction process of the virtual human system according to an embodiment of the present invention.

[0055] Figure 17 This is a schematic diagram of the thought process for virtual human intent recognition in an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] like Figure 1 As shown, this embodiment discloses an interactive system in a human-machine-virtual human mode, including: a data acquisition module for acquiring multimodal data; the multimodal data includes: voice data, text data, image data, or video stream data; an intent recognition module for performing intent recognition on the multimodal data and obtaining intent recognition results; intent recognition includes: code word recognition, question-and-answer database recognition, and large language model recognition; and a virtual human module for recognizing the user's identity before acquiring the multimodal data, inputting the identity information as prompt words into the large language model to adjust the interaction mode, and streaming the intent recognition results in a target format on the virtual human page.

[0059] Specifically, the core of this invention lies in using virtual humans as anthropomorphic operational agents for complex systems. The virtual human system not only assists users in completing various tasks but also enhances the user's interactive experience by simulating human interaction methods (such as voice, facial expressions, and gestures). The system's design allows users to interact with complex backend systems through natural communication, thereby simplifying operational processes and improving efficiency.

[0060] The virtual human system employs a multimodal data fusion inference engine based on Large Language Models (LLM), supporting various input methods such as voice, text, images, and video streams. This inference engine integrates technologies such as speech recognition, visual recognition, and natural language understanding, enabling it to synthesize various input information, deduce user needs, and format them into commands that the system can directly process. This design significantly improves the response speed and processing power of complex systems, making user interaction with complex systems more intuitive and efficient.

[0061] The virtual human system employs a model alignment mechanism on both the input and output sides. On the input side, the system converts information such as speech, facial features, and gestures into text, then provides input to the LLM (Local Language Model) through prompt engineering. On the output side, the system abstracts the system, modules, applications, and capabilities of the "full-stack platform" into "tools" or "capabilities," organizes the call elements through text extraction functionality of the large language model, and formats the output. This alignment mechanism ensures efficient matching and smoothness between the system's input and output.

[0062] This invention also includes controlling remote machines by installing a remote machine "Trojan" (the system encapsulates common tools as services and packages them into executable programs, which are then installed on controllable machines). This technology enables the virtual human system to perform tasks across devices, rather than being limited to single-device operation, thus realizing a "human-machine-virtual human-large system" interaction mode of "multi-device" interconnection. For example, the subsystems of the full-stack platform with the virtual human's own screen open can be pushed to a computer connected to the large screen via voice or gesture commands, or to a client terminal for more complex mouse and keyboard interactions.

[0063] Furthermore, the data acquisition module includes: a speech recognition submodule, used to acquire the audio stream of the user's device, and use a preset volume threshold and silence duration to determine the end of the user's speech input. After the user's speech input ends, audio features are extracted and acoustic model decoding is performed on the audio stream to obtain text information, i.e., the speech recognition result; and a hot word replacement submodule, used to replace the speech recognition result with a preset hot word list to obtain the target text information; the hot word list includes proper nouns and special words.

[0064] Specifically, such as Figure 15-17 As shown, speech recognition includes:

[0065] Browser-side audio stream acquisition: The speech recognition module obtains audio streams through the browser.

[0066] The `navigator.mediaDevices.getUserMedia()` API retrieves the user's audio stream from their device, thereby capturing the user's microphone input in the webpage and enabling real-time voice data acquisition. The acquired audio stream is then transmitted in real-time to the audio processing module for further processing.

[0067] Audio Processing and Silence Detection: The speech recognition module intelligently determines the end of user voice input by using a volume threshold and silence duration. After each audio data acquisition, the system checks the current audio volume based on a set volume threshold (volumeThreshold). If the volume is below the threshold, it is considered to have entered a silence phase. If the silence duration exceeds a set threshold (silenceThreshold, e.g., 1500 milliseconds), the system automatically terminates the current voice recording and sends the recording data to the backend for processing.

[0068] WebSocket Real-Time Bidirectional Communication and Streaming Recognition: Recorded data is transmitted to the speech recognition engine via the WebSocket protocol. The speech recognition engine can convert the speech stream into text in real time. Its recognition process includes audio feature extraction, acoustic model decoding, etc., and finally outputs the recognized text.

[0069] The speech recognition module enables low-latency voice data transmission via WebSocket connection, establishing a persistent bidirectional communication channel between the client and server. After a successful connection, the system begins capturing microphone data and formatting the voice into an appropriate data format (such as an Int16Array), sending it in batches to the speech recognition engine for recognition. During transmission, the system continuously monitors the connection status and processes and provides feedback promptly after the recognition results are generated.

[0070] Once the speech recognition results are generated, the system returns the recognized text to the client in batches via WebSocket. The recognition results (including speech text, recognition model type, and timestamps) are pushed to the Unity3D engine in real time in JSON format. The Unity3D engine then streams the recognized text as "subtitles" on the virtual human page, thus completing the full streaming real-time speech recognition process.

[0071] Hotword Replacement: To improve recognition accuracy and personalization, the system supports intelligent replacement of speech recognition results based on a user-configured hotword list. The hotword library includes common user-defined proper nouns (such as personal names and place names) and special words. The system will automatically replace words appearing in the real-time recognition results to ensure that the final output text meets user expectations while avoiding misrecognition caused by accents or unclear speech.

[0072] Furthermore, the intent recognition module includes: a first intent recognition submodule, used to perform code word recognition on the target text information. When the user intent is determined by recognizing a preset code word or trigger keyword, the task processing link is matched, and the systems and applications registered on the full-stack platform are automatically invoked. When the code word recognition fails, the target text information is subjected to question-and-answer database recognition. The target text information is converted into a vector and compared with the question vector of the question-and-answer database. The compared question vector is output, the task processing link is matched, and the systems and applications registered on the full-stack platform are automatically invoked. When the semantic similarity does not exceed a preset similarity threshold, the target text information is subjected to large language model recognition. The target text information is input into the large language model. When it is detected that a backend tool needs to be invoked, the structured data of the function call request is output, the function is executed, and the execution result is returned to the model or sent back to the user in the form of a response message.

[0073] The backend tool can be a function, such as getting the current time and date, or it can be an API of a full-stack platform system and application, such as "querying inventory data" or "opening a system".

[0074] Backend tools are typically Python code, packaged as a standalone program, responsible for the actual calls. In most cases, the main tools can be considered to primarily call the API interfaces provided by the systems and applications registered by the full-stack platform.

[0075] Furthermore, outputting the compared question vector includes: converting the target text information into a query vector, querying the most matching semantic tag from the vector database of the question-and-answer library according to the vector similarity, mapping the system and application registered by the full-stack platform, and using the vector similarity as a matching score to output a candidate list sorted in reverse order of matching score.

[0076] Furthermore, the intent recognition module also includes: a second intent recognition submodule, used to perform a similarity search using knowledge vectors in the vector database when the large language model fails to detect the backend tool, obtain the text segment most relevant to the independent segment, replace placeholders with the most relevant text segment and the independent segment to generate a prompt word, input the prompt word into the large language model, detect the backend tool to be called, output the structured data of the function call request, execute the function, and return the execution result to the model or back to the user in the form of a response message; wherein, the vector database slices the original text information to obtain several independent segments, each of the independent segments retains overlap information according to a preset overlap field, and stores the overlapped information.

[0077] Specifically, such as Figure 16 As shown, the intent recognition module includes: the intent recognition module implementation in the backend system, encapsulated as a backend service using the Python Flask framework. This is mainly achieved through three methods.

[0078] Code-based approach: By recognizing predefined codes or trigger keywords, the system determines user intent and efficiently matches task processing links. For example, when a user says "accept task," the system triggers the task flow through keyword matching instead of a natural language question-and-answer process, and automatically calls the recommendation module to recommend systems, applications, modules, and blocks (mini-programs) registered on the full-stack platform.

[0079] Based on a question-and-answer database: The question-and-answer database provides configurable predefined question-and-answer pairs. Questions are vectorized and stored in a vector database, enabling the system to perform semantic matching and retrieval based on user questions. Administrators can store frequently asked questions and their answers in the database through a backend interface. When a user asks a question, the system converts it into a vector and compares it with the question vectors in the database. Based on the calculated semantic similarity and a set threshold, the system determines the most matching answer and returns it to the user. When the similarity exceeds the threshold, the system directly provides the corresponding answer.

[0080] The answer is defined as a formatted text that the virtual human can "understand," including the reply text and corresponding actions. For example, "Q: I want to watch a large model video. A: Okay, here's the large model video. [Video] large model video.mp4," the virtual human can synthesize speech to reply with the text "Okay, here's the large model video," and then play "large model video.mp4" on the virtual human system screen. This method requires no model training and can respond to user needs instantly. In addition, the system periodically inputs customized question-and-answer data in batches into the large model fine-tuning process to train a unique task processing model.

[0081] Based on Large Language Models (LLMs): When building intelligent dialogue systems or task processing workflows, Large Language Models (LLMs) can infer user intent from natural language commands and accordingly select and invoke appropriate backend systems, tools, or APIs. This process typically utilizes the function call functionality of the LLM. Function calls are an extension capability of LLMs, enabling the model to not only generate natural language text but also output structured call requests. For example, when the model identifies a specific type of task (such as querying a database, invoking computing tools, or executing backend services) during a dialogue with a user, it automatically generates parameters and requests in a predefined function call format and passes them to the relevant system. The principle is as follows:

[0082] Embedded function descriptions: During model development or application integration, these descriptions provide the large model with function definitions and descriptions, including the function name, required parameters, parameter types, and return value descriptions. These descriptions are often injected through system or developer message hints, ensuring the model is fully aware of the callable functions.

[0083] Contextual Inference and Matching: After receiving user input, the large model determines the task type and required information processing method based on the dialogue context and pre-trained knowledge. If the model determines that a backend tool needs to be called to fulfill the user's request, it will proactively output structured data (such as JSON format) of a function call request, including the function name and required parameters.

[0084] Function execution and result return: Once a function call request is generated, the system layer is responsible for actually executing the function (e.g., querying the database, calling external APIs, etc.). After execution, the result is returned to the model or directly to the user as a response message (often containing structured data or text descriptions).

[0085] Feedback and follow-up dialogue: After obtaining the function results, the model will continue to understand and process the returned information, and then respond to the user in natural language to ensure that the user receives complete answers, analysis results or suggestions.

[0086] Large Language Model Retrieval Enhancement (RAG) Service Module: This module is the core of the virtual human's ability to intelligently answer user questions. By combining large language models with vector databases, document parsing, text slicing, and other techniques, this module can efficiently retrieve, process, and reason about knowledge, thereby providing answers that incorporate business knowledge and reducing the illusion of large language models. This module ensures that the system provides answers based on relevant, validated knowledge, rather than solely on the generative capabilities of the language model.

[0087] Vector Database: The vector database is the physical carrier of the knowledge base. It enables efficient natural language retrieval based on vector similarity. The system converts document content into vector representations (equivalent to the representation of knowledge in a high-dimensional space) and stores these vectors in the vector database. When a user submits a query, the vector database's vector retrieval service can compare the input vector (converted by the word embedding model) with the knowledge vectors already stored in the database, using vector similarity (Euclidean distance, cosine similarity) as a metric, and retrieve the most relevant text fragments corresponding to the knowledge vectors from the file system, thereby efficiently finding relevant text knowledge in massive amounts of information.

[0088] Document parsing: To handle documents from different sources and formats, the system supports various document parsing modules through the langchain framework and OCR models. The task of document parsing is to convert different types of documents (such as PDFs, Word documents, plain text files, etc.) into structured text data. This process includes steps such as text extraction and format standardization to ensure that searchable and inference-worthy content is extracted from the original documents.

[0089] Text Slicing and Vectorization: For long documents, to avoid information overload or exceeding the model's processing capacity and to improve retrieval efficiency, this module slices the document content. Text slicing involves dividing long text into smaller, independent segments, each containing a certain amount of information and possessing independent semantic meaning. Some overlap is also preserved between each text segment to ensure that contextual information is not lost. This module uses an extension class of `ChineseRecursiveTextSpliter` based on the `langchain` framework to implement comprehensive segmentation based on length and punctuation. These sliced ​​texts are vectorized using a word embedding model, and the vectors, along with the original text, the path to the source file, and the text's position within the source file, are stored in a vector database. The original file is then stored in the file system for easy retrieval and to improve the reliability of inference.

[0090] Knowledge Base Management Module: In addition to the RAG process, this module includes a fully GUI-enabled knowledge base management system with file management capabilities. Users can input documents via the interface, API, or ETL pipeline. The management system automatically parses, segments, and vectorizes the document content, storing it in a vector database to build a continuously updated knowledge base. When a user submits a query, this module uses a word embedding model to vectorize the user's question and performs a similarity search using the knowledge vectors stored in the vector database to retrieve the most relevant text fragments. These retrieved text fragments provide the necessary information foundation for subsequent reasoning.

[0091] The knowledge base management module assembles prompt words and directs the large model for inference: This module combines the recalled text fragments with the user's query to generate a prompt word that includes the query context and relevant knowledge. As follows:

[0092] The instruction answers the question based on known information. If the answer cannot be obtained from the information, please reply "Unsure". Answers should be concise, accurate, and logically clear, prioritizing reliable content directly from the known information. Fabricated elements are not allowed in the answer, and answers should be in Chinese. This includes: known information {{context}} and the question {{question}}. This module will replace the placeholder {{context}} with relevant text blocks and the placeholder {{question}} with the user's question text. It will then be sent to the large model for inference. Recommendation and invocation of full-stack platform capabilities: Semantic enhancement based on a large language model: Combining context, this module uses the large model to enhance user semantics and correct inaccurate speech recognition text, ensuring that the user's voice input instructions are clearer, more accurate, and more purposeful before being passed to subsequent steps.

[0093] Recommendation System: The recommendation system uses a vector database to maintain a list of semantic matching tags, function descriptions, input parameter lists, parameter value specifications, system routes, and other metadata for the full-stack platform's systems, applications, modules, and blocks (mini-programs), forming a "capability registry" for the full-stack platform. Each record contains a "profile" of the tool.

[0094] The recommendation system transforms the user's input text into query vectors, retrieves the most matching semantic tags from the vector database based on vector similarity, and traces back to a candidate list of systems, applications, modules, and components (mini-programs) across the entire platform through mapping. Simultaneously, a reranking model is used to re-rank and score the descriptions of the retrieved systems or tools against the user's input text, using the similarity score as the matching value, and outputting a candidate list ranked inverted by matching score to recommend to the user.

[0095] Key information extraction based on the large language model: The large language model extracts the corresponding input parameters based on the input parameter field list of the system, application, module, and block (mini-program) of the full-stack platform, checks whether the extracted values ​​meet the specifications, and then returns the results to the Unity side for rendering, taking the top k items for the user to choose from.

[0096] Capability invocation on a full-stack platform: Capability invocation depends on the full-stack platform's systems, applications, modules, and components (mini-programs). These components must: 1. Have a web-based UI. 2. Be able to be directly routed to their sub-interfaces via "deep linking". 3. Be able to parse and respond to parameters registered in the "input parameter list" of the recommendation system.

[0097] The candidate list returned by the recommendation system includes the "deep links" and "parameters" of the system, application, module, and component (mini-program) across the entire platform, which are then embedded in the virtual human page via iframe for rendering. Users can then directly perform subsequent operations by touching the virtual human screen, or use voice or gesture commands to push the screen to a remote machine for magnified display, or use a regular client computer for mouse and keyboard control.

[0098] Furthermore, the virtual human module includes: a face recognition submodule, used to acquire video streams from cameras, extract facial images from the video streams, obtain facial feature vectors, compare the facial feature vectors with the facial features of registered users stored in the face database, calculate the similarity between the facial feature vectors and facial features, and when the similarity of the comparison results exceeds the facial feature threshold, face recognition is successful, and a list of user identity IDs matching the facial features is returned; and an interaction adjustment submodule, used to merge relevant identity information based on the user identity ID list into speech, provide feedback information to the user, and input the identity information as prompt words into the large language model to adjust the interaction method.

[0099] Specifically, the sensing module includes:

[0100] Face Detection: This module utilizes "human-like" perception technology to activate the virtual human system. When the system detects a face through the camera, it automatically initiates a dialogue process. This process is integrated into the Unity engine (see Human-Computer Interaction Process), leveraging the Unity and OpenCV frameworks to support camera video stream capture. Video stream capture uses Unity's WebCamTexture class, while face detection relies on OpenCV's CascadeClassifier. The face detection model uses the Haar classifier by default, which supports asynchronous loading and allows its parameters, such as scaleFactor and minSize, to be adjusted through configuration. Furthermore, the system design is flexible and can be replaced with deep neural networks (DNNs) or other advanced detection algorithms as needed.

[0101] Video stream acquisition and device access: The face recognition module obtains the video stream from the camera through the navigator.mediaDevices.getUserMedia() API provided by the browser. The video stream is then transmitted from the client to the server for processing by the subsequent face recognition module.

[0102] Facial Feature Extraction, Identity Matching, and Similarity Calculation: The system extracts a unique facial feature vector from each facial image using a facial recognition model. The extracted facial feature vector is compared with the facial features of registered users stored in the face database. Deep learning-based algorithms (such as cosine similarity calculation or Euclidean distance) are used to evaluate the similarity between the currently detected face and the facial features of each user in the face database. When the similarity score of the comparison result exceeds the system threshold (configurable), the system considers the recognition successful and returns a list of user IDs (sorted by similarity from highest to lowest) that match the facial features.

[0103] Identity Information Query and Feedback: After successfully matching a user's identity, the system will query the database for relevant identity information based on the returned user ID. This information includes, but is not limited to, the user's name, job title, age, gender, profile picture, and other personal information and personalized settings. The system will then convert this identity information into natural language and synthesize it into speech to provide the user with personalized feedback. For example, "Hello, Staff Officer Zhang," so that the user can perceive that their identity has been confirmed.

[0104] Furthermore, identity information is also used as a user identification cue, input into the large language model so that the system can "recognize" and "remember" the user's identity, and then adjust the interaction method or content based on the user's identity characteristics. This allows the system to combine user identity information to generate more accurate and personalized responses or recommendations.

[0105] The virtual human program will open variables in memory to remember the ID of the person being recognized by facial recognition, and then retrieve the person's information from the database using the ID and present it to the large model.

[0106] Once the virtual human successfully identifies a person, it is equivalent to starting a "conversation". At this time, "remembering" is equivalent to locking the current user. When the user says "goodbye" to it in a voice conversation, the virtual human will end the "conversation" and start identifying new users.

[0107] Furthermore, the virtual human module also includes: a virtual rendering submodule, used to stream the intent recognition results as text in the target format on the virtual human page; and a speech synthesis submodule, used to convert the text of the intent recognition results into speech and obtain the speech synthesis results.

[0108] Specifically, such as Figure 2-3 As shown, the overall technical solution of the system includes:

[0109] The virtual human system is designed based on a B / S architecture, using modern mainstream programming languages ​​and frameworks to ensure flexibility and scalability. The specific technology stack includes: programming languages: HTML, CSS, JavaScript, C#, Python; frameworks: AntDv, Vue, Unity, WebGL, Flask, Docker, Conda; storage: MySQL, HDFS, Faiss; core middleware: OpenCV, XInference, Vllm, Nginx; application layer network protocols: HTTP, WebSocket; and AI models.

[0110] The UI design of the virtual human system includes:

[0111] like Figure 4 As shown, the virtual human interface uses the Unity3D engine and is rendered on the web page based on WebGL.

[0112] The backend system includes:

[0113] like Figure 5 As shown, the question-answering library contains special question input and output configurations, which can expand the system output without training a model.

[0114] like Figure 6 As shown, the prompt library includes the management and configuration of prompt words.

[0115] like Figure 7-9 As shown, the voice library includes virtual human voices, configuration and management of voice control commands, and model parameter tuning.

[0116] like Figure 10 As shown, the image library contains the configuration and management of virtual human images, actions, scenes, etc.

[0117] like Figure 11 As shown, the perception library includes functions such as registration management of facial and pose biometric information, and model parameter tuning.

[0118] like Figure 12-13 As shown, the knowledge base includes the management of knowledge related to virtual humans. It utilizes RAG to achieve comprehensive knowledge-based reasoning for large-scale models.

[0119] like Figure 14 As shown, the model set includes: a large language model and its supporting smaller models.

[0120] Furthermore, the speech synthesis submodule includes: a speech synthesis unit, which is used to convert the text of the intent recognition result into speech, obtain the speech synthesis result, correct the pronunciation of polyphonic words in the speech synthesis result, and comprehensively segment according to punctuation marks and length, and then broadcast the speech synthesis text in a queue format.

[0121] A voice interruption unit, which is used to pause or stop the current voice output and start playing new synthesized voice content if the user issues a new command during the process of playing the synthesized voice result.

[0122] Specifically, the technical implementation of the virtual human system includes:

[0123] Real-time streaming speech synthesis module: Real-time streaming speech synthesis (TTS, Text-to-Speech) is a key module for the virtual human system to output to users for interaction. The speech synthesis module is not just static speech synthesis. It supports real-time processing of voice output, including quick response to user input and interruption management during audio playback.

[0124] Streaming speech synthesis mainly includes the following key elements: Real-time conversion of text to speech: Convert the real-time generated text information into an audio stream and transmit it to the client in real time through WebSocket. Control and playback of the audio stream: The system not only generates speech but also needs to control the playback state of the audio in real time according to user interaction. Interruption and recovery of speech playback: When the user issues a new command or request, the current speech playback can be interrupted.

[0125] Streaming speech synthesis process: The process of streaming speech synthesis requires highly optimizing the speech generation and transmission of the system to ensure low latency and high-quality output during the speech synthesis process. The specific process is as follows:

[0126] Processing of user input: After the synthesis system receives the input, it will correct the pronunciation of multi-syllable words for different models, which can be configured by the administrator. That is, through "typo" replacement, such as replacing "种" with "重", to ensure that the pronunciation is natural and correct.

[0127] Text segmentation and speech synthesis: Each paragraph or sentence will be synthesized separately and converted into audio data. For long texts, the system supports splitting the text into multiple parts and generating speech data one by one. The system performs segmentation processing on the text, splitting it comprehensively according to punctuation marks and length. The segmentation processing can avoid system lag caused by overly long text in a single request.

[0128] Speech synthesis engine: The audio synthesis process is carried out by the background speech synthesis (TTS) engine, and each text is generated in real time using the KanTTS model in sequence.

[0129] Output and playback of real-time audio stream: The synthesized audio will be transmitted to the client device in real time. Through the WebSocket protocol, the audio data will be immediately sent and played in a stream. On the client device, the audio playback is passed to the Unity audio playback component, etc., and starts playing immediately.

[0130] Voice interruption and interruption handling: During the system's voice playback, if the user issues a new command or an event occurs that requires interrupting the current voice (such as the user asking a new question or making a request), the system will immediately pause or stop the current voice output and start playing new voice content.

[0131] The interruption mechanism in speech synthesis consists of two parts: interruption detection: real-time monitoring of system input (such as new speech recognition results or UI interactions). When new input is detected, the system decides whether to interrupt the current speech playback.

[0132] Interruption Handling: Once an interruption is decided, the system will pause the current speech playback and immediately stop audio output. If it is new text or a command, the system will restart the synthesis and playback of the new speech stream.

[0133] Furthermore, the system also includes:

[0134] The remote machine control module encapsulates client operations as API services, packages them into a standalone program, installs it on the remote machine, displays the full-stack platform on the remote machine, captures the interface state of the remote machine, and controls the remote machine based on the interface state.

[0135] Specifically, the remote machine control module includes: This system encapsulates common client operations into HTTP or WebSocket-based API services, packages them as standalone programs, and installs them on the target machine to achieve automated control of remote devices via the network. This enables cross-device collaborative operation, as illustrated in the following classic example:

[0136] Browser Automation Control: This module launches the browser through an encapsulated browser operation service and automatically loads the specified URL. At the same time, it passes the necessary information of the full-stack platform's system, applications, modules, and blocks (mini-programs) through the URL's query parameters, including the user's original question text, tool definitions, and parameters extracted through large model processing. This allows the full-stack platform's system, applications, modules, and blocks (mini-programs) opened in the virtual human screen to be opened on remote devices, achieving a "push" effect.

[0137] In addition, the remote machine control module records the browser page opening status of each remote device and uses WebSocket to capture the interface status of the remote machine and return it to the virtual human system. It uses the Visual Large Language Model (VLM) to "understand" the browser status of the controlled machine, such as whether the browser is focused, whether the browser has an open tab for a certain full-stack platform, the currently focused tab, etc. Users can give voice commands to the virtual human to perform other operations on the browser in the remote device, such as focusing, closing a tab, etc.

[0138] PowerPoint Automation: The system further extends to the automation control of office software, enabling remote control of PowerPoint presentations through the encapsulation of specific operation libraries. This system can perform operations such as pausing, page turning, and closing during PPT playback, and provides an API interface as a service, allowing users to initiate calls from remote devices to control the PPT playback status. This feature is suitable for remote meetings, online training, and presentation scenarios.

[0139] Split-screen display and page rendering: The system uses the PyQt framework for web page rendering and multi-screen display management, supporting flexible display of web page content on large screens or multiple display devices. Through split-screen technology, the system can display different web pages or application content in parallel within the same display environment. This feature allows users to efficiently manage and operate multiple tasks on remote machines, improving display efficiency and operational flexibility.

[0140] Model Inference Service: The following is an inference solution for the AI ​​model required by a system based on an open-source model (verified in a practically implemented virtual human system). Upgrades are available based on actual needs and results.

[0141] Large model inference cluster: The virtual human system can use the general large language model GLM-4-Chat-9B and the visual large language model GLM-4V-9B.

[0142] Inference platform: XInference is used on x86+Nvidia architecture machines. MindIE is used on domestic ARM+Ascend machines. Both inference frameworks employ VLLM and are deployed on multiple instances.

[0143] Load balancing: Use Nginx to proxy multiple large model instances and employ a least connection strategy for load balancing. Configure Nginx's `limit_conn` to limit the maximum concurrency to ensure stability.

[0144] Small model inference cluster: Word embedding models can use bge-large-zh-v1.5, with a multi-instance CPU inference service built on the XInference platform. Reranking models can use bge-reranker-v2-m3, with a multi-instance CPU inference service built on the XInference platform. Speech recognition (ASR) models can use FunASR, with inference services deployed via Docker. Text-to-speech (TTS) models can use KAN-TTS and CosyVoice, deployed via Conda, and an inference service with interruption support based on Python WebSocket is developed in-house. Face recognition models can use Compreface, with inference services deployed via Docker. Pose recognition models can use Mediapipe, deployed via Conda, and a service based on Python WebSocket is developed in-house.

[0145] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. An interactive system in a human-machine-virtual human mode, characterized in that, include: The data acquisition module is used to acquire multimodal data; The multimodal data includes: voice data, text data, image data, or video stream data; An intent recognition module is used to perform intent recognition on the multimodal data and obtain intent recognition results; the intent recognition includes: code word recognition, question-and-answer database recognition, and large language model recognition; The virtual human module is used to identify the user's identity before acquiring the multimodal data, input the identity information as prompt words into the large language model to adjust the interaction mode, and stream the intent recognition results as text in the target format on the virtual human page.

2. The interactive system in the human-machine-virtual human mode according to claim 1, characterized in that, The data acquisition module includes: The speech recognition submodule is used to acquire the audio stream of the user device and determine the end of the user's speech input by using a preset volume threshold and silence duration. After the user's speech input ends, the audio stream is subjected to audio feature extraction and acoustic model decoding to obtain text information, i.e., speech recognition result. The hot word replacement submodule is used to replace the speech recognition results with a preset hot word list to obtain target text information; the hot word list includes proper nouns and special words.

3. The interactive system in the human-machine-virtual human mode according to claim 2, characterized in that, The intent recognition module includes: The first intent recognition submodule is used to perform code word recognition on the target text information. When the user intent is determined by recognizing a preset code word or triggering keyword, the task processing link is matched, and the system and application registered on the full-stack platform are automatically called. When the code word recognition fails, the target text information is subjected to question-and-answer database recognition. The target text information is converted into a vector and compared with the question vector of the question-and-answer database. The compared question vector is output, the task processing link is matched, and the system and application registered on the full-stack platform are automatically called. When the semantic similarity does not exceed the preset similarity threshold, the target text information is subjected to large language model recognition. The target text information is input into the large language model. When it is detected that a backend tool needs to be called, the structured data of the function call request is output, the function is executed, and the execution result is returned to the model or sent back to the user in the form of a response message. The response message is displayed through the virtual human's UI interface, showing text information, character motion rendering, and voice broadcast, and is used to reply to the user.

4. The interactive system in the human-machine-virtual human mode according to claim 3, characterized in that, The intent recognition module further includes: outputting the compared question vector, including: The target text information is converted into a query vector. The most matching semantic tag is retrieved from the vector database of the question-and-answer library according to the vector similarity. This maps the system and application registered by the full-stack platform. At the same time, the vector similarity is used as a matching score, and a candidate list is output in reverse order of matching score.

5. The interactive system in the human-machine-virtual human mode according to claim 4, characterized in that, The intent recognition module also includes: The second intent recognition submodule is used to perform a similarity search using knowledge vectors in the vector database when the large language model fails to detect the backend tool. This search retrieves the text fragment most relevant to the independent fragment, replaces placeholders with the most relevant text fragment and the independent fragment to generate a prompt word, inputs the prompt word into the large language model, detects the backend tool to be called, outputs the structured data of the function call request, executes the function, and returns the execution result to the model or back to the user in the form of a response message. The vector database slices the original text information to obtain several independent segments. Each independent segment retains overlap information according to a preset overlap field, and the overlapped information is stored.

6. The interactive system in the human-machine-virtual human mode according to claim 1, characterized in that, The virtual human module includes: The face recognition submodule is used to acquire the video stream from the camera, extract facial images from the video stream, obtain facial feature vectors, compare the facial feature vectors with the facial features of registered users stored in the face database, calculate the similarity between the facial feature vectors and the facial features, and when the similarity of the comparison results exceeds the facial feature threshold, the face recognition is successful, and a list of user identity IDs that match the facial features is returned. The interaction adjustment submodule is used to merge relevant identity information from the user identity ID list into speech, provide feedback information to the user, and input the identity information as prompt words into the large language model to adjust the interaction method.

7. The interactive system in the human-machine-virtual human mode according to claim 1, characterized in that, The virtual human module also includes: The virtual rendering submodule is used to stream the intent recognition results as text in the target format onto the virtual human page. The speech synthesis submodule is used to convert the text of the intent recognition result into speech and obtain the speech synthesis result.

8. The interactive system in the human-machine-virtual human mode according to claim 7, characterized in that, The speech synthesis submodule includes: The speech synthesis unit is used to convert the text of the intent recognition result into speech, obtain the speech synthesis result, correct the pronunciation of polyphonic words in the speech synthesis result, and segment it according to punctuation marks and length. After segmentation, the speech synthesis text is broadcast in a queue. The voice interruption unit is used to pause or stop the current voice output and start playing new voice-synthesized content if the user issues a new command during the playback of the voice synthesis result.

9. The interactive system in the human-machine-virtual human mode according to claim 1, characterized in that, The system also includes: The remote machine control module is used to encapsulate client operations as API services, package them as standalone programs, install them on a remote machine, display the full-stack platform on the remote machine, and control the remote machine.

Images