A visual and large model-based simulated digital human real-time intelligent voice interaction system and method thereof

Abstract

The application relates to a visual and large model-based simulation digital human real-time intelligent voice interaction system and a method thereof, and aims to solve the problems of inaccurate target speaker recognition and high response delay in digital human voice interaction in a complex scene. The system effectively identifies the range through a camera ring, triggers audio collection in combination with face detection, locks the target speaker by using lip movement recognition and sound image fusion technology and reduces noise, generates an answer by means of a large language model (LLM) and knowledge retrieval enhancement (RAG) technology after the voice is converted into text, generates low-delay voice by using vLLM accelerated voice synthesis technology, drives a preloaded digital human image to synthesize a video stream and pushes the video stream to a front end for real-time rendering. The application realizes accurate sound pickup, low-delay interaction and fast switching of the digital human image in a complex environment, improves the accuracy and real-time performance of intelligent voice question and answer, and is suitable for scenes such as government offices and exhibition halls.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence language interaction technology, specifically a real-time intelligent voice interaction system and method for simulated digital humans based on vision and large models. Background Technology

[0002] With the widespread application of artificial intelligence (AI) technology across various industries, "AI+" has become a key force driving economic and social development. Faced with fierce market competition and rapidly changing customer demands, enterprises are seeking to reduce labor and time costs through AI technology, enabling more efficient operation and management, and providing humanized and personalized service experiences to meet the diverse needs of the public. With the development of AI technologies such as large-scale modeling and speech synthesis, enterprises are leveraging digital human products such as digital employees and virtual human IPs to provide efficient, convenient, and personalized services, further achieving cost reduction, efficiency improvement, and promoting digital transformation. However, the inability to accurately identify the target speaker's audio in complex scenarios such as government service halls and exhibition halls is one of the major problems facing today's real-time voice interaction systems for digital humans. Especially when interacting with simulated digital humans in real-time, issues such as inaccurate speech recognition and low end-to-end real-time response efficiency arise. Therefore, how to accurately identify the target speaker's voice in complex scenarios and improve the end-to-end response efficiency of simulated digital humans is a current hot research topic.

[0003] In recent years, with the development of AI large-scale model technology, research on digital humans at home and abroad has become increasingly diversified. Patent CN114463827A discloses a multimodal real-time emotion recognition method and system based on DS evidence theory. It utilizes speech text and audio signals combined with facial features for emotion recognition, effectively improving the accuracy of emotion recognition. Patent CN120048276A discloses a real-time voice dialogue method and system based on LLM technology applied to an operations and maintenance platform. It uses speech recognition and LLM technology to perform semantic analysis on text data to identify the intent of user commands, effectively addressing the problems of high response latency and low accuracy in real-time voice dialogue on operations and maintenance platforms. Patent CN119806332A discloses a proactive AI digital human interaction method and system based on visual analysis. By analyzing user actions, it generates solution plans, improving the intelligence, real-time nature, and user experience of the interaction. Patent CN119520933A discloses an AI digital human broadcasting method and application platform based on multimodal fusion. It utilizes multimodal data such as voice, text, and images to fuse and extract digital human voice broadcasting content, enhancing the user's viewing experience. Patent CN118411989A discloses a real-time voice dialogue device and method that allows AI speech to be interrupted by voice. Compared to existing technologies, this improves the user experience of real-time voice dialogue and reduces the consumption of unnecessary resources. Patent CN118052918A discloses a virtual digital human application method based on AI multimodal interaction, integrating multimodal modeling, artificial intelligence speech recognition, image recognition, big data, and other technologies to achieve a virtual digital human application based on AI multimodal interaction. The aforementioned research has addressed issues such as latency and emotional expression in simple multimodal interaction of digital humans based on AI technologies such as visual recognition and multimodal models. However, breakthroughs are still needed to achieve integrated "perception-cognition-drive" real-time voice interaction of simulated digital humans with low latency and high robustness in noisy, multi-person dialogue scenarios.

[0004] Focusing on the pain points of accurately parsing the target speaker's voice in complex scenarios and the end-to-end real-time response of the simulated digital human, this paper proposes a real-time intelligent voice interaction system and method for simulated digital humans based on vision and large models. Summary of the Invention

[0005] The purpose of this invention is to overcome the above-mentioned technical problems and provide a real-time intelligent voice interaction system and method for simulated digital humans based on vision and large models. This method integrates AI technologies such as visual detection, large models, RAG and speech synthesis. After detecting people through a camera, it automatically records their voices, locks the speaker through lip movement recognition and audio-visual fusion, and then converts the speech to text after noise reduction and wake-up. It generates answers by combining LLM+RAG and accelerates the streaming synthesis of low-latency speech based on the vLLM framework. It also combines digital human generation technology to drive the real-time rendering of digital human video streams, realizing high-quality intelligent voice question-and-answer service around the clock.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a real-time intelligent voice interaction system for simulated digital humans based on visual perception and large model driving, comprising:

[0007] The camera is used to capture images of the interactive area. The effective recognition range of the area captured by the front-end digital human camera can be drawn through the digital human management terminal.

[0008] Microphone, used for audio capture;

[0009] Front-end devices are used for the visual display and audio playback of digital humans;

[0010] The personnel detection and audio acquisition triggering module is connected to the camera and microphone. It is used to detect whether a face enters the effective recognition range. When a face is detected, the microphone is automatically triggered to start for audio acquisition.

[0011] The target recognition module and speech recognition and preprocessing module connect the personnel detection and audio acquisition triggering module and the target recognition module. They are used to identify the target interaction object from the faces that have entered the effective recognition range, use lip movement recognition technology and audio-visual fusion technology to determine whether the target interaction object is speaking, and use audio noise reduction algorithm to preprocess the audio signal of the target interaction object.

[0012] Voice wake-up and conversion module: Connects to the speech recognition and preprocessing module, used for wake-up word recognition, and performs speech-to-text processing on the audio information of the recognized wake-up word to generate text information;

[0013] Semantic analysis and response module: Connected to the voice wake-up and conversion module, used to receive text information, perform semantic analysis and retrieval on the text information, and generate response text;

[0014] Speech Synthesis and Output Module: Connects to the semantic analysis and response module, and is used to perform real-time speech synthesis of the response text by using the vLLM-accelerated Cosyvoice speech model and combining it with the reference timbre of specified timbre features, and output audio files;

[0015] Digital Human Generation and Push Module: Connects to the speech synthesis and output module, used to synthesize videos based on the synthesized speech and pre-loaded digital human images, and push them to the front-end page in real time through the streaming media server;

[0016] The front-end rendering module connects the digital human generation and push module and the front-end device, and is used to obtain the synthesized audio and video from the streaming media server in real time and perform real-time rendering.

[0017] A real-time intelligent voice interaction method for simulated digital humans based on visual perception and large model-driven approaches: characterized by:

[0018] S1. To effectively identify people who want to talk to the digital human, the effective recognition range is delineated in the shooting area of ​​the front-end digital human camera through the digital human management terminal.

[0019] S2. When a person is detected entering the effective recognition range, the system calls the face detection algorithm to determine whether a face has been detected. If a face is detected, the microphone is automatically turned on to collect audio. Otherwise, the camera image is acquired to detect the person.

[0020] S3. Use face detection algorithm to identify target interaction object, use lip recognition technology and audio-visual fusion technology to determine whether the target interaction object is speaking. If speaking, use audio noise reduction algorithm to preprocess the audio signal of the target speaker. If not speaking, the system continues to detect.

[0021] S4. Identify the wake word in the noise-reduced audio information. If the wake word is identified, convert the audio signal into text information through the speech-to-text model. If it is not identified, prompt the user to say the wake word.

[0022] S5. Based on the large language model and RAG technology, perform semantic understanding and knowledge retrieval on the input text information to generate the answer text;

[0023] S6. The response text is segmented and then enters the queue buffer management. The vLLM is used to accelerate the Cosyvoice speech model and the reference timbre with specified timbre features is combined to perform high-concurrency speech synthesis on the text segments.

[0024] S7. Synthesize video based on the synthesized speech and pre-loaded digital human image, and push it to the front-end page in real time through the streaming media server;

[0025] S8. Acquire synthesized audio and video frames from the streaming media server in real time and render the page in real time. The front-end page will then display the images and present a highly realistic digital human image and voice interaction effect.

[0026] Further, step S3 includes:

[0027] S31. Detect face information in the image based on the YOLO5-Face detection algorithm, reduce the number of face boxes by adjusting the similarity threshold, calculate the size of the face boxes, select the face box with the largest area and record the face features. If the face information detected twice in a row is less than the specified threshold, the face information is taken as the target interaction object; otherwise, continue to acquire image information to determine the target interaction object.

[0028] S32. After identifying the target interaction object, extract the facial information of the target object, and then use the 68-point dlib model to detect lip key points. Calculate the degree of lip opening in each frame to obtain the lip movement intensity sequence LipSeq, and then perform short-term variance detection to obtain the visual VAD. Six consecutive frames with variance > 0.05 are considered "start", and 12 silent frames are considered "end".

[0029] S33. The visual VAD and voice VAD are fused using the SyncScore. When the SyncScore > 0.7, it is determined that the face is speaking in the current time period.

[0030] S34. After confirming that the target object has made a sound, call the audio noise reduction model to perform front-end processing on the original audio, filter out environmental noise, system echo and other non-speech interference, and output pure audio with high signal-to-noise ratio.

[0031] Further, step S4 includes:

[0032] S41. Determine the current recognition status. If it is already "awakened", directly send the audio output in S34 into the ASR module to transcribe the text. If it is still "not awake", enable the wake word detection model. When the confidence exceeds the threshold, immediately update the status to "awakened" and seamlessly connect the audio and subsequent streams to the ASR. Otherwise, prompt the user to say the wake word and maintain the original status.

[0033] S42. Use the FunASR-FangYan model, which is fine-tuned based on dialect, to convert audio to text.

[0034] Furthermore, after speech-to-text conversion in step S4, the qwen2.5-14B large model is used to determine the user's intent in the text information and to identify whether image information needs to be combined. If the intent is determined to require image information, the multimodal large model is called and combined with the Prompt prompt to parse the current frame image. The parsing result is then combined with the text information according to the rules, and the text information is finally output. Otherwise, the text information is output directly.

[0035] Furthermore, in step S5, the system obtains the speech-to-text result, uses the Embedding algorithm to convert the text into a numerical vector to process semantic information, combines knowledge bases such as vector libraries and knowledge graphs, and uses the cosine similarity vector matching algorithm to retrieve relevant semantic information. Finally, the large language model, combined with prompt words, generates a more suitable answer.

[0036] Further, in step S5, the bge-large-zh-v1.5 vectorization model is used to vectorize the text results, converting the text into numerical vectors to process semantic information. Then, the cosine similarity algorithm is used to retrieve semantic information that exceeds a predefined threshold.

[0037] Furthermore, in step S5, the retrieved semantic information is summarized using the qwen2.3-14B large model combined with prompt words.

[0038] Furthermore, in step S7, the audio files of the speech synthesis are managed using a queue. At the same time, the digital human image is preloaded into memory. Then, multi-process video frame processing is performed. This process uses a multi-process parallel mechanism to process the audio files. Each process contains multiple threads, and each thread executes the "frame generation" and "frame rendering" operations in sequence. Finally, the video frames are synthesized into a video. Then, the processed video is put into a "pushing queue". A multi-process and event-mode listening strategy is used to monitor whether the pushing queue is empty. If it is not empty, the synthesized video stream is pushed in real time via RTC. This uses the SRS pushing service framework combined with FFmpeg pushing commands to realize the real-time pushing of the digital human. If it is empty, the default digital human video is pushed in real time.

[0039] Furthermore, the multi-process video frame processing in step S7 includes:

[0040] Image preloading: Adopting the singleton design pattern combined with a multi-process optimization strategy, the digital human image material is preloaded into memory through resource preloading and caching mechanisms;

[0041] Frame generation: Based on the lightweight diffusion U-Net generation algorithm combined with digital human image materials, lip-sync image frames are generated from audio frames;

[0042] Frame rendering, rendering only occurs on the mouth area. First, a bounding box is calculated using facial key points, and the area is cropped to 256×256 and fed into the network. The network output is then resized back to the original bounding box size, and the system performs temporal smoothing on the bounding box sequence.

[0043] Frame compositing involves buffering the original video frames one by one, then using Poisson technology to blend the rendered mouth parts back to their corresponding positions. If the audio is longer than the video, the system will reverse the original frame sequence and play it again for seamless extension. Finally, FFmpeg repackages the composite frames and the original audio to generate a complete video with lip-sync.

[0044] The beneficial effects of this invention are as follows: This invention achieves accurate sound pickup in complex environments based on visual detection and audio-visual fusion technology. By combining visual detection and audio-visual fusion technology, it achieves accurate localization and audio extraction of the target speaker. Furthermore, it effectively processes the target audio through advanced audio noise reduction algorithms, solving the problem that in noisy environments such as exhibition halls, conference rooms, and government service halls, intelligent question answering is prone to introducing environmental noise and the voices of nearby non-target speakers, resulting in inaccurate speech recognition results or the inclusion of irrelevant content.

[0045] This invention introduces streaming processing optimization, heterogeneous computing power collaborative scheduling, and multi-process parallel mechanism to reconstruct the simulated digital human voice interaction process, realize low-latency and high-concurrency intelligent dialogue capabilities, and significantly improve the user interaction experience.

[0046] This invention employs a singleton design pattern combined with a multi-process optimization strategy to reconstruct the digital human character loading process. By using resource preloading and caching mechanisms, it trades space for time, significantly improving the response speed of character switching and the overall system performance. This effectively solves the problems of long simulation character switching time and negative impact on user experience. Attached Figure Description

[0047] Figure 1 It is a flowchart of real-time intelligent voice interaction for simulated digital humans based on vision and large models;

[0048] Figure 2 This is a flowchart of the process for automatically activating the microphone based on a facial recognition algorithm;

[0049] Figure 3 This is a flowchart of the speech-to-text process based on audio-visual fusion, lip-sync recognition, and large-model technology.

[0050] Figure 4 This is a flowchart of the streaming query results return based on a large language model and RAG technology;

[0051] Figure 5 This is a flowchart of high-concurrency streaming speech synthesis based on vLLM inference acceleration technology;

[0052] Figure 6 This is a flowchart of the real-time generation process of digital human audio and video.

[0053] Figure 7 This is a block diagram of a real-time intelligent voice interaction system for simulated digital humans based on vision and large models. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0055] Embodiments of the present invention: such as Figure 7 As shown, a real-time intelligent voice interaction system for simulated digital humans based on visual perception and large model-driven architecture includes:

[0056] The camera is used to capture images of the interactive area. The effective recognition range of the area captured by the front-end digital human camera can be drawn through the digital human management terminal.

[0057] Microphone, used for audio capture;

[0058] Front-end devices are used for the visual display and audio playback of digital humans;

[0059] The personnel detection and audio acquisition triggering module is connected to the camera and microphone. It is used to detect whether a face enters the effective recognition range. When a face is detected, the microphone is automatically triggered to start for audio acquisition.

[0060] The target recognition module and speech recognition and preprocessing module connect the personnel detection and audio acquisition triggering module and the target recognition module. They are used to identify the target interaction object from the faces that have entered the effective recognition range, use lip movement recognition technology and audio-visual fusion technology to determine whether the target interaction object is speaking, and use audio noise reduction algorithm to preprocess the audio signal of the target interaction object.

[0061] Voice wake-up and conversion module: Connects to the speech recognition and preprocessing module, used for wake-up word recognition, and performs speech-to-text processing on the audio information of the recognized wake-up word to generate text information;

[0062] Semantic analysis and response module: Connected to the voice wake-up and conversion module, used to receive text information, perform semantic analysis and retrieval on the text information, and generate response text;

[0063] Speech Synthesis and Output Module: Connects to the semantic analysis and response module, and is used to perform real-time speech synthesis of the response text by using the vLLM-accelerated Cosyvoice speech model and combining it with the reference timbre of specified timbre features, and output audio files;

[0064] Digital Human Generation and Push Module: Connects to the speech synthesis and output module, used to synthesize videos based on the synthesized speech and pre-loaded digital human images, and push them to the front-end page in real time through the streaming media server;

[0065] The front-end rendering module connects the digital human generation and push module and the front-end device, and is used to obtain the synthesized audio and video from the streaming media server in real time and perform real-time rendering.

[0066] A real-time intelligent voice interaction method for simulated digital humans based on visual perception and large model-driven approaches includes:

[0067] 1. Delineate the effective recognition range of the camera: In order to effectively identify people who want to talk to the digital human, the administrator needs to delineate the effective recognition range of the front-end digital human camera's shooting area through the digital human management terminal;

[0068] II. (e.g.) Figure 2 (As shown) When a person is detected entering the effective recognition range, the system calls the face detection algorithm to determine whether a face has been detected. If a face is detected, the microphone is automatically turned on to collect audio; otherwise, the system continues to acquire camera footage for person detection.

[0069] III. (as) Figure 3 (As shown) The system enables speech-to-text conversion through audio-visual fusion, lip-reading recognition, and large-scale model technology: After the microphone is turned on, the system simultaneously collects image and audio data through the camera and microphone. First, it uses a face detection algorithm to identify the target interaction object. Then, it analyzes the lip movements of the target object based on a lip movement recognition algorithm. Combined with audio-visual matching fusion technology, it determines whether the target object is speaking. If speaking, it uses an audio noise reduction algorithm to preprocess the audio signal of the target speaker. If not speaking, the system continues to detect. After audio noise reduction, it attempts to identify the wake word in the audio. If the wake word is identified, the audio signal is converted into text information through the speech-to-text model. If not identified, the user is prompted to say the wake word. Subsequently, the system determines whether it needs to combine image information. If not, it directly outputs text information. If so, it uses a multimodal large-scale model combined with the Prompt prompt to combine the image parsing results with the text information according to rules, and finally outputs the text information.

[0070] Specifically:

[0071] 1) Identify the target interaction object. Detect facial information in the image using the YOLO5-Face detection algorithm (once per second). Reduce the number of face bounding boxes by adjusting the similarity threshold. Simultaneously calculate the size of each face bounding box (as shown in Formula 1). Select the face bounding box with the largest area and record its facial features. If two consecutive detections show that the facial information is less than the specified threshold (as shown in Formula 2), then that face information is considered the target interaction object. Otherwise, continue acquiring image information to determine the target interaction object.

[0072]

[0073]

[0074]

[0075] 2) Lip movement recognition algorithm: After identifying the target interaction object, extract the target object's facial information, and then use a 68-point dlib model to detect lip key points (index 48-67 for the outer lip). Calculate the degree of lip opening in each frame to obtain the lip movement intensity sequence LipSeq, and then perform short-time variance (as in formula 4) detection to obtain the visual VAD (lip movement start and end). Six consecutive frames with variance > 0.05 are considered "start", and 12 silent frames are considered "end".

[0076]

[0077]

[0078] Where t represents time, and 49, 62, ..., 67 are the indexes of lip key points.

[0079] 3) Voice-image matching and fusion technology: The SyncScore is used to fuse visual VAD and voice VAD. When the SyncScore > 0.7, it is determined that the face is speaking in the current time period.

[0080]

[0081] 4) Audio noise reduction and wake-up: After confirming that the target has spoken, the audio noise reduction model is called to process the original audio in the front end, filtering out environmental noise, system echo and other non-speech interference, and outputting clean audio with high signal-to-noise ratio; then the current recognition status is judged. If it is already "wake-up", the audio output in S34 is directly sent to the ASR module to transcribe the text; if it is still "not wake-up", the wake word detection model is enabled. When the confidence exceeds the threshold, the status is immediately updated to "wake-up" and the audio frame and subsequent streams are seamlessly connected to ASR; otherwise, the user is prompted to say the wake word and the original status is maintained.

[0082] 5) Audio-to-text conversion: After acquiring audio segments, the FunASR-FangYan model, fine-tuned based on dialect, is used to convert audio into text. Compared to the open-source FunASR model, the accuracy of dialect speech recognition is improved by 5%.

[0083] 6) Based on the large model, implement intent recognition to determine whether to combine image information. Use the qwen2.5-14B large model to judge the user intent in the text information and determine whether it needs to be combined with image information. If the intent is determined to be combined with image information, call the multimodal large model and combine it with the Prompt prompt to parse the current frame image. Combine the parsing result with the text information according to the rules and finally output the text information. Otherwise, directly output the text information.

[0084] Text message example:

[0085] 1. Directly output text information: {

[0086] User question: Please introduce China Unicom.

[0087] }

[0088] 2. Combining image analysis results with text information: {

[0089] User question: What am I holding in my hand?

[0090] Image analysis results: There is a man wearing a pojo shirt in the picture, holding a laptop in his hand.

[0091] }

[0092] IV. (e.g.) Figure 4 (As shown) Based on the large language model and RAG technology, the system returns query results in a streaming manner: The system obtains speech-to-text results, uses the Embedding algorithm to convert the text into numerical vectors to process semantic information, combines knowledge bases such as vector libraries and knowledge graphs, and uses the cosine similarity vector matching algorithm to retrieve relevant semantic information. Finally, the large language model, combined with prompt words, generates a more suitable answer.

[0093] Specifically:

[0094] 1) Vectorize the text and perform vector matching. Use the bge-large-zh-v1.5 vectorization model to transform the text results into numerical vectors to process semantic information. Then use the cosine similarity algorithm to retrieve semantic information that exceeds the predefined threshold (e.g., Formula 6).

[0095]

[0096] Where A and B are two n-dimensional vectors, and θ is a predefined threshold.

[0097] 2) Summary of large language model: The semantic information retrieved is summarized by using the qwen2.3-14B large model in combination with prompt words.

[0098] Example of a Prompt prompt: {

[0099] You are a voice Q&A assistant, capable of analyzing user input and executing the following capabilities: if the user's input does not contain image parsing results, execute capability 1; otherwise, execute capability 2.

[0100] User input: #Text result

[0101] Capability 1: Able to answer user questions based on user input and knowledge base context. Please use Chinese for your thought process and answers; do not use English or other languages. Do not include website information in your output. If the answer cannot be obtained from the knowledge base context, you can answer the user's question from a professional perspective, with the total word count not exceeding 100 characters.

[0102] Ability 2: Able to answer user questions based on user input and image analysis results. Please use Chinese for your thought process and answers; do not use English or other languages. The output should not contain website information, and the total word count should not exceed 50 characters.

[0103] V. (e.g.) Figure 5 (As shown) High-concurrency streaming speech synthesis is achieved based on vLLM inference acceleration technology: Based on the text results returned by the large language model, the text is then segmented according to preset rules such as "by sentence or semantic unit". The segmented text segments are put into queue buffer management to ensure the order of processing. Then, vLLM is used to accelerate the Cosyvoice large speech model and combine it with the reference timbre of the specified timbre features to perform high-concurrency speech synthesis on the text segments and output audio files. After testing, the speech synthesis speed is improved by about 60% under high concurrency.

[0104] VI. (e.g.) Figure 6(As shown) Real-time audio-visual generation of a multi-process digital human is implemented based on audio: To ensure the orderly playback of the simulated digital human's voice, the audio files for speech synthesis are managed using a queue. Simultaneously, the digital human image is preloaded into memory. Then, a multi-process parallel mechanism is used to process the audio files. To improve audio processing speed, each process contains multiple threads. Each thread sequentially executes "frame generation" and "frame rendering" operations, finally synthesizing the video frames into a video. The processed video is then entered into a "pushing queue." The system checks if the pushing queue is empty; if not, the synthesized video stream is pushed in real-time via RTC; otherwise, the default digital human video is pushed in real-time. The final output is a "real-time digital human image."

[0105] Specifically:

[0106] 1) Multi-process video frame processing: In order to improve the video synthesis speed, the digital human image is preloaded first and called directly when the image needs to be switched. Then, multiple processes are used to process multiple audio files or multiple audio files in parallel. At the same time, in order to make full use of computing resources and improve processing efficiency, each process contains two threads. The threads cooperate with each other to generate ordered video frames. Finally, ffmpeg is used to synthesize the video and put it into the streaming queue to wait for streaming.

[0107] Image preloading: By adopting the singleton design pattern and multi-process optimization strategy, digital human image materials are preloaded into memory through resource preloading and caching mechanisms; by trading space for time, the response speed of image switching and the overall system performance are significantly improved.

[0108] Frame generation: Based on the lightweight diffusion U-Net generation algorithm combined with digital human image materials, lip-sync image frames are generated from audio frames;

[0109] Frame rendering, rendering only occurs on the mouth area. First, a bounding box is calculated using facial key points, and the area is cropped to 256×256 and sent to the network. The network output is then resized back to the original bounding box size. The system performs temporal smoothing on the bounding box sequence, and the rest of the facial area remains unchanged.

[0110] Frame compositing involves caching the original video frames one by one, then using Poisson technology to blend the rendered mouth parts back to their corresponding positions. If the audio is longer than the video, the system will reverse the original frame sequence and play it again for seamless extension, avoiding abrupt jumps and saving storage. Finally, FFmpeg repackages the composite frames and the original audio to generate a complete video with lip-sync.

[0111] 2) RTC real-time push streaming: By designing a multi-process and event-mode listening strategy to monitor whether the push streaming queue is empty, and using the SRS push streaming service framework in combination with FFmpeg push streaming commands to realize the real-time push streaming of the digital human, the final output is "real-time digital human video".

[0112] VII. Real-time rendering of digital human visualization on the front end: Synthesized audio and video frames are acquired from the streaming media server in real time and rendered on the page in real time. The visualization is then displayed on the front end, providing users with a realistic, intelligent, and visual interactive service experience.

Claims

1. A real-time intelligent voice interaction system for simulated digital humans based on visual perception and large model-driven architecture, characterized in that: include: The camera is used to capture images of the interactive area. The effective recognition range of the area captured by the front-end digital human camera can be drawn through the digital human management terminal. Microphone, used for audio capture; Front-end devices are used for the visual display and audio playback of digital humans; The personnel detection and audio acquisition triggering module is connected to the camera and microphone. It is used to detect whether a face enters the effective recognition range. When a face is detected, the microphone is automatically triggered to start for audio acquisition. The target recognition module and speech recognition and preprocessing module connect the personnel detection and audio acquisition triggering module and the target recognition module. They are used to identify the target interaction object from the faces that have entered the effective recognition range, use lip movement recognition technology and audio-visual fusion technology to determine whether the target interaction object is speaking, and use audio noise reduction algorithm to preprocess the audio signal of the target interaction object. Voice wake-up and conversion module: Connects to the speech recognition and preprocessing module, used for wake-up word recognition, and performs speech-to-text processing on the audio information of the recognized wake-up word to generate text information; Semantic analysis and response module: Connected to the voice wake-up and conversion module, used to receive text information, perform semantic analysis and retrieval on the text information, and generate response text; Speech Synthesis and Output Module: Connects to the semantic analysis and response module, and is used to perform real-time speech synthesis of the response text by using the vLLM-accelerated Cosyvoice speech model and combining it with the reference timbre of specified timbre features, and output audio files; Digital Human Generation and Push Module: Connects to the speech synthesis and output module, used to synthesize videos based on the synthesized speech and pre-loaded digital human images, and push them to the front-end page in real time through the streaming media server; The front-end rendering module connects the digital human generation and push module and the front-end device, and is used to obtain the synthesized audio and video from the streaming media server in real time and perform real-time rendering.

2. A real-time intelligent voice interaction method for simulated digital humans based on visual perception and large model-driven approaches: characterized by: S1. Use the digital human management terminal to delineate the effective recognition range of the front-end digital human camera's shooting area; S2. When a person is detected entering the effective recognition range, the system calls the face detection algorithm to determine whether a face has been detected. If a face is detected, the microphone is automatically turned on to collect audio. Otherwise, the camera image is acquired to detect the person. S3. Use face detection algorithm to identify target interaction object, use lip recognition technology and audio-visual fusion technology to determine whether the target interaction object is speaking. If speaking, use audio noise reduction algorithm to preprocess the audio signal of the target speaker. If not speaking, the system continues to detect. S31. Detect face information in the image based on the YOLO5-Face detection algorithm, reduce the number of face boxes by adjusting the similarity threshold, calculate the size of the face boxes, select the face box with the largest area and record the face features. If the face information detected twice in a row is less than the specified threshold, the face information is taken as the target interaction object; otherwise, continue to acquire image information to determine the target interaction object. S32. After identifying the target interaction object, extract the facial information of the target object, and then use the 68-point dlib model to detect lip key points. Calculate the degree of lip opening in each frame to obtain the lip movement intensity sequence LipSeq, and then perform short-term variance detection to obtain the visual VAD. Six consecutive frames with variance > 0.05 are considered "start", and 12 silent frames are considered "end". S33. The visual VAD and voice VAD are fused using the SyncScore. When the SyncScore > 0.7, it is determined that the face is speaking in the current time period. S34. After confirming that the target has made a sound, call the audio noise reduction model to perform front-end processing on the original audio, filter out environmental noise, system echo and other non-speech interference, and output pure audio with high signal-to-noise ratio. S4. Identify the wake word in the noise-reduced audio information. If the wake word is identified, convert the audio signal into text information through the speech-to-text model. If it is not identified, prompt the user to say the wake word. S5. Based on the large language model and RAG technology, perform semantic understanding and knowledge retrieval on the input text information to generate the answer text; S6. The response text is segmented and then enters the queue buffer management. The vLLM is used to accelerate the Cosyvoice speech model and the reference timbre with specified timbre features is combined to perform high-concurrency speech synthesis on the text segments. S7. Based on the synthesized speech and the preloaded digital human image, a video is synthesized and pushed to the front-end page in real time through a streaming media server. Specifically, the audio files for speech synthesis are managed using a queue. At the same time, the digital human image is preloaded into memory. Then, multi-process video frame processing is performed. It adopts a multi-process parallel mechanism to process the audio files. Each process contains multiple threads. Each thread executes the "frame generation" and "frame rendering" operations in sequence. Finally, the video frames are synthesized into a video. Then, the processed video is put into the "pushing queue". A multi-process and event-mode listening strategy is used to monitor whether the pushing queue is empty. If it is not empty, the synthesized video stream is pushed in real time through RTC. It uses the SRS pushing service framework combined with FFmpeg pushing commands to realize the real-time pushing of the digital human. If it is empty, the default digital human video is pushed in real time. S8. Acquire synthesized audio and video frames from the streaming media server in real time and render the page in real time. The front-end page will then display the images and present a highly realistic digital human image and voice interaction effect.

3. The method according to claim 2, characterized in that: Step S4 includes: S41. Determine the current recognition status. If it is already "awakened", directly send the audio output in S34 to the ASR module to transcribe the text. If it is still "not awake", enable the wake word detection model. When the confidence exceeds the threshold, immediately update the status to "awakened" and seamlessly connect the audio and subsequent streams to ASR. Otherwise, prompt the user to say the wake word and maintain the original status. S42. Use the FunASR-FangYan model, which is fine-tuned based on dialect, to convert audio to text.

4. The method according to claim 2, characterized in that: In step S4, after speech-to-text conversion, the qwen2.5-14B large model is used to determine the user's intent in the text information and whether it is necessary to combine it with image information. If the intent is determined to be to combine it with image information, the multimodal large model is called and combined with the Prompt prompt to parse the current frame image. The parsing result is then combined with the text information according to the rules, and the text information is finally output. Otherwise, the text information is output directly.

5. The method according to claim 4, characterized in that: In step S5, the system obtains the speech-to-text result, uses the Embedding algorithm to convert the text into a numerical vector to process semantic information, combines knowledge bases such as vector libraries and knowledge graphs, and uses the cosine similarity vector matching algorithm to retrieve relevant semantic information. Finally, the large language model, combined with prompt words, generates a more suitable answer.

6. The method according to claim 5, characterized in that: In step S5, the bge-large-zh-v1.5 vectorization model is used to vectorize the text results, converting the text into numerical vectors to process semantic information. Then, the cosine similarity algorithm is used to retrieve semantic information that exceeds a predefined threshold.

7. The method according to claim 5, characterized in that: In step S5, the semantic information retrieved is summarized using the qwen2.3-14B large model combined with prompt words.

8. The method according to claim 2, characterized in that: Step S7, multi-process video frame processing, includes: Image preloading: Adopting the singleton design pattern combined with a multi-process optimization strategy, the digital human image material is preloaded into memory through resource preloading and caching mechanisms; Frame generation: Based on the lightweight diffusion U-Net generation algorithm combined with digital human image materials, lip-sync image frames are generated from audio frames; Frame rendering, rendering only occurs on the mouth area. First, a bounding box is calculated using facial key points, and the area is cropped to 256×256 and fed into the network. The network output is then resized back to the original bounding box size, and the system performs temporal smoothing on the bounding box sequence. Frame compositing involves buffering the original video frames one by one, then using Poisson technology to blend the rendered mouth parts back to their corresponding positions. If the audio is longer than the video, the system will reverse the original frame sequence and play it again for seamless extension. Finally, FFmpeg repackages the composite frames and the original audio to generate a complete video with lip-sync.

Images