Real-time visual digital human interrogation method and system based on webrtc and llm and medium

Abstract

The application discloses a real-time visual digital human inquiry method and system based on WebRTC and LLM, and a medium. The method comprises the following steps: first, receiving an inquiry task containing mode selection; second, executing a differentiated process according to the selected mode: if it is a semi-automatic mode, automatically conducting a conversation between a digital human image and an inquired party based on a structured question list generated in advance based on a large language model; if it is a manual mode, conducting a real-time conversation between a digital human image and an inquired party based on real-time voice of a human inquirer. Finally, after the conversation ends, a multi-dimensional structured summary report is automatically generated by using a large language model according to complete interaction text. Through the dual-mode design and intelligent analysis, the application solves the problems of low efficiency of traditional manual inquiry, the awkwardness of face-to-face conversation between a visual inquirer and an inquired party, the unfavorable atmosphere for the company, and the time-consuming report arrangement, and significantly improves the standardization degree of inquiry.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and information processing technology, and in particular to a real-time visual digital human interrogation method, system and medium based on WebRTC and LLM. Background Technology

[0002] In recent years, the rapid development of artificial intelligence technology has driven the widespread application of digital humans in fields such as remote consultation, customer service, and education and training. Traditional remote video conferencing systems mainly rely on manual operation, resulting in problems such as low efficiency, high cost, and insufficient standardization. With the maturity of Web Real-Time Communication (WebRTC) technology, real-time audio and video communication can provide low-latency and highly reliable interactive support, while advancements in Large Language Models (LLM) have provided powerful capabilities for natural language processing and structured data analysis. However, existing digital human video conferencing systems still have shortcomings in terms of intelligent interaction and business adaptability.

[0003] Currently, most video interrogation systems only support a single manual or automatic mode, lacking flexible semi-automated interaction mechanisms and making it difficult to dynamically adjust interrogation strategies according to different business needs. For example, in standardized business scenarios, relying entirely on manual interrogation leads to inefficiency, while fully automated interrogation may lack the flexibility to handle complex issues. Furthermore, existing systems also have shortcomings in question generation and interaction optimization, such as the inability to dynamically adjust questioning strategies based on session context, or difficulty in ensuring smooth interaction and data integrity under network fluctuations.

[0004] Meanwhile, traditional video conferencing systems typically rely on manual processing for session record analysis and report generation, resulting in low efficiency and a tendency to miss key information. Although some systems have attempted to incorporate natural language processing (NLP) technology for text analysis, the lack of deep integration with real-time interaction often leads to reports with insufficient structure, failing to meet the needs of business decision-making. Therefore, there is an urgent need for a digital human-based video conferencing method that combines WebRTC real-time communication capabilities with LLM intelligent analysis technology to improve interaction efficiency, optimize question generation strategies, and achieve automated, structured analysis of session data. Summary of the Invention

[0005] To address at least one of the aforementioned technical problems, this invention proposes a real-time visual digital human interrogation method, system, and medium based on WebRTC and LLM.

[0006] The first aspect of this invention provides a real-time visual digital human interrogation method based on WebRTC and LLM, comprising:

[0007] The video query management platform receives video query tasks from the business initiator and determines the video query mode based on the video query tasks. The video query mode includes a semi-automatic video query mode and a manual video query mode.

[0008] When the visual inquiry mode is a semi-automatic visual inquiry mode, a structured question list is constructed, and the digital human automatically asks questions based on the structured question list, thus forming a semi-automatic visual inquiry strategy.

[0009] When the video interrogation mode is the human video interrogation mode, a real-time interactive session is established, the real-time interactive session is processed by voice, and the digital human asks questions based on the voice-processed interactive session, thus forming a human video interrogation strategy.

[0010] Obtain the conversation interaction text after the semi-automatic or manual interrogation strategy ends, perform structured transformation on the conversation interaction text based on LLM, and generate a structured analysis report.

[0011] In this solution, the step of receiving the video query task from the business initiator through the video query management platform and determining the video query mode based on the video query task includes a semi-automatic video query mode and a manual video query mode, specifically as follows:

[0012] Based on the visual query management platform, visual query instruction data from the business initiator is obtained, and a visual query task is created according to the visual query instruction data, including task identifier, visual query mode marking information, and visual query target object information;

[0013] The query mode of the business initiator is determined based on the query mode selection information of the query task.

[0014] In this solution, when the interrogation mode is a semi-automatic interrogation mode, the construction of a structured question list is specifically as follows:

[0015] The platform obtains unstructured text from the business initiator based on the video inquiry management platform. The unstructured text includes the background of the video inquiry dialogue and core objective information. The background of the video inquiry dialogue includes the type of dialogue event and the event description. The core objective information includes details of the event to be verified, points of contention, and request information.

[0016] The semantic parsing engine extracts the dialogue event type and event description from the unstructured text as the first feature vector, and extracts the details of the event to be verified and the points of contention as the second feature vector. The first and second feature vectors are input into the prompt engineering generation model. The prompt engineering generation model encodes the dialogue scene of the first feature vector to generate scene constraints through a multi-head attention mechanism, and decomposes the target intent of the second feature vector to generate target refinement conditions.

[0017] Based on the scenario constraints and target refinement conditions, a structured instruction prompt template is constructed. The structured instruction prompt template includes an event background description section, a core question generation instruction section, and an answer format requirement section.

[0018] The structured instruction prompt template is input into the instruction parsing layer of the LLM. The hierarchical decoder of the instruction parsing layer sequentially generates an event background summary, a list of key questions, and a list of optional follow-up questions. Each question item in the list of key questions is associated with a corresponding point of contention weight coefficient.

[0019] The list of key issues is prioritized based on the weight coefficients of the points of contention, and a structured list of issues is generated.

[0020] In this solution, the automatic questioning operation based on the structured question list by the digital human constitutes a semi-automatic visual inquiry strategy, specifically as follows:

[0021] The digital human platform is launched and a WebRTC video session channel is established. The virtual digital human image is rendered in real time through the video session channel. When the business initiator and the interviewee successfully connect to the video session, the first question text in the current structured question list is obtained.

[0022] The audio acquisition device of the viewed party is turned off, and the question text is converted into a digital human broadcast voice stream using text-to-speech technology. At the same time, the question text is rendered onto the subtitle display area of ​​the user interface. The network fluctuation during the digital human broadcast process is monitored in real time, and the data packet loss rate of the current transmission link is calculated based on the network fluctuation.

[0023] When the data packet loss rate exceeds the first threshold, the encoding bitrate of the digital human video stream is reduced. Based on the time point when the data packet loss rate is greater than the preset loss rate, the broadcast voice data segment with transmission quality lower than the preset quality is determined, and the voice data segment is cached in the local buffer.

[0024] When the data packet loss rate falls below the second threshold, the untransmitted key interaction segments are extracted from the local buffer and retransmitted. The key interaction segments include the problem broadcast voice stream and the corresponding subtitle synchronization timestamp.

[0025] After the digital human finishes broadcasting the voice stream, the audio acquisition device of the interviewee is activated and the real-time speech recognition service is started. When the duration of the acquired audio of the interviewee's response exceeds the preset silence detection threshold, the audio of the response is transcribed into the response text in real time based on the speech recognition service, and the response text is updated to the response subtitle box. At the same time, the audio data packet loss rate during the transcription process is monitored.

[0026] When the audio data packet loss rate exceeds the third threshold, the current speech recognition process is interrupted and the person being questioned is prompted to repeat the answer. At the same time, the transcribed text fragments are spliced ​​together with the subsequently recorded audio of the repeated answer, and the time axis is aligned.

[0027] When the person being questioned clicks the "complete answer" command or the duration of silence in the answer audio reaches a preset termination threshold, the audio acquisition device is turned off and the current question answer text is recorded. At the same time, the semantic matching degree between the answer text and the corresponding question is calculated.

[0028] When the semantic matching degree is lower than the preset matching threshold, follow-up questions are extracted from the structured question list and inserted into the current question sequence; otherwise, the next question text is obtained according to the order of the structured question list.

[0029] The interactive loop of repeatedly executing digital human question broadcasting, data packet loss rate monitoring, audio acquisition of the queried party, lost data recovery, and semantic matching degree judgment continues until all questions in the structured question list have been interacted with, then the video session is terminated and the question and answer record text is output, thus forming a semi-automatic video query strategy.

[0030] In this solution, when the interrogation mode is a human interrogation mode, a real-time interactive session is established, the real-time interactive session is processed for speech, and the digital human asks questions based on the speech-processed interactive session, thus forming a human interrogation strategy, specifically as follows:

[0031] A real-time video interactive session channel is established based on WebRTC, and the business initiator of the real-time video interactive session channel is rendered as a virtual digital human image in real time based on the digital human platform.

[0032] Real-time voice data of the business initiator and the interviewee is acquired, the real-time voice data is transcribed and the digital human voice is driven to broadcast.

[0033] When the initiator or the party being interviewed issues a command to end the interview control, the interactive record text is output, thus constituting a manual interview strategy.

[0034] In this solution, the real-time acquisition of real-time voice data from the service initiator and the queryee, the transcription of the real-time voice data, and the driving of digital human voice playback specifically involve:

[0035] The original voice streams of the service initiator and the queryee are captured based on the WebRTC real-time video interactive session channel, and the original voice streams are transmitted in real time to the voice conversion service for voiceprint feature conversion processing to generate a converted voice stream with a uniform timbre.

[0036] The converted speech stream is sent back to the digital human platform, which generates digital lip-sync control commands and speech broadcast signals to drive the virtual digital human to perform real-time speech output and lip-sync.

[0037] The automatic speech recognition engine converts the converted speech stream and the original speech stream of the interviewee into conversational interactive text in real time, and renders the conversational interactive text into the subtitle area in real time.

[0038] In this solution, the step of obtaining the conversation interaction text after the semi-automatic or manual interrogation strategy has ended, and then performing a structured transformation on the conversation interaction text based on LLM to generate a structured analysis report, specifically involves:

[0039] Obtain the conversation interaction text after the semi-automatic or manual interrogation strategy ends, and send the conversation interaction text to the LLM by calling the interrogation aggregation interface based on the digital human platform;

[0040] The conversational interaction text is structured using LLM based on advanced prompting engineering techniques to construct a structured analysis report;

[0041] The structured analysis report, conversational text, and audio / video data are compiled into a data package and sent back to the visual management platform for storage.

[0042] In this solution, the step of using LLM to structurally process the conversational interaction text based on advanced prompting engineering techniques and constructing a structured analysis report specifically involves:

[0043] Based on LLM, the event operation flow description of the queried party in the conversation interaction text is extracted, the stage to which the event operation flow description belongs is identified, and a behavior description summary containing the event operation flow and the stage to which it belongs is generated.

[0044] Obtain standard operating procedure data for the event, compare the standard operating procedure data with the behavior description summary, and identify the event operation risk points of the queried party, including deviation from the event operation procedure and operation errors.

[0045] The behavioral description summary and event operation risk points are imported into the LLM to generate a recommendation report text, including information to be reviewed, event operation improvement suggestions, and processing time limit suggestions;

[0046] The behavioral description summary, event operation risk points, and suggestion report text are used to generate a structured analysis report.

[0047] A second aspect of the present invention also provides a real-time visual interrogation digital human interrogation system based on WebRTC and LLM. The system includes: a memory and a processor. The memory includes a real-time visual interrogation digital human interrogation method program based on WebRTC and LLM. When the real-time visual interrogation digital human interrogation method program based on WebRTC and LLM is executed by the processor, it implements the steps of the real-time visual interrogation digital human interrogation method based on WebRTC and LLM as described in any of the above claims.

[0048] A third aspect of the present invention also provides a computer-readable storage medium comprising a real-time visual digital human interrogation program based on WebRTC and LLM, wherein when the real-time visual digital human interrogation program based on WebRTC and LLM is executed by a processor, it implements the steps of the real-time visual digital human interrogation method based on WebRTC and LLM as described in any of the preceding claims.

[0049] This invention discloses a real-time visual inquiry method, system, and medium based on WebRTC and LLM. The method includes: first, receiving an inquiry task that includes mode selection; second, executing differentiated processes according to the selected mode: in semi-automatic mode, an automated conversation is conducted between the digital human avatar and the inquirer based on a pre-generated structured question list from a large language model; in human mode, the digital human avatar conducts a real-time conversation with the inquirer based on real-time voice input from a human inquirer. Finally, after the conversation ends, a multi-dimensional structured summary report is automatically generated using the large language model based on the generated complete interactive text. This invention, through dual-mode design and intelligent analysis, solves the problems of low efficiency in traditional human inquiry, the awkwardness of face-to-face conversations between the inquirer and the inquirer, the negative impact on company harmony, and the time-consuming report compilation, significantly improving the standardization of inquiry. Attached Figure Description

[0050] Figure 1 A flowchart of a real-time visual digital human interrogation method based on WebRTC and LLM according to the present invention is shown;

[0051] Figure 2 A flowchart of the semi-automatic interrogation mode of the present invention is shown;

[0052] Figure 3 A flowchart of the manual interrogation mode of the present invention is shown;

[0053] Figure 4 A block diagram of a real-time visual interrogation digital human interrogation system based on WebRTC and LLM is shown. Detailed Implementation

[0054] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0055] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0056] Figure 1 A flowchart of a real-time visual digital human interrogation method based on WebRTC and LLM according to the present invention is shown.

[0057] like Figure 1 As shown, the first aspect of the present invention provides a real-time visual digital human interrogation method based on WebRTC and LLM, comprising:

[0058] The video query management platform receives video query tasks from the business initiator and determines the video query mode based on the video query tasks. The video query mode includes a semi-automatic video query mode and a manual video query mode.

[0059] When the visual inquiry mode is a semi-automatic visual inquiry mode, a structured question list is constructed, and the digital human automatically asks questions based on the structured question list, thus forming a semi-automatic visual inquiry strategy.

[0060] When the video interrogation mode is the human video interrogation mode, a real-time interactive session is established, the real-time interactive session is processed by voice, and the digital human asks questions based on the voice-processed interactive session, thus forming a human video interrogation strategy.

[0061] Obtain the conversation interaction text after the semi-automatic or manual interrogation strategy ends, perform structured transformation on the conversation interaction text based on LLM, and generate a structured analysis report.

[0062] It's worth noting that by receiving and parsing video inquiry tasks from business initiators through the video inquiry management platform, the system can intelligently select the most suitable video inquiry mode based on actual business needs. This retains the flexibility of manual video inquiry while providing the efficiency of semi-automatic video inquiry. In semi-automatic video inquiry mode, the system implements a standardized questioning process by constructing a structured question list. The digital human automatically asks questions based on this list, which not only improves interaction efficiency but also ensures that key questions are not overlooked. Furthermore, a smart sorting mechanism prioritizes core questions. In manual video inquiry mode, by establishing real-time interactive conversations and processing voice, the digital human can naturally and fluently present human-assigned questions, maintaining the realism of interpersonal interaction while providing the standardized advantages of digital human interaction. Finally, through intelligent analysis and processing of the conversational text, leveraging the powerful semantic understanding capabilities of LLM, unstructured dialogue content is transformed into structured analysis reports. This significantly improves information extraction efficiency, automatically identifies key information, and generates professional business suggestions, significantly enhancing the intelligence and execution efficiency of the video inquiry process while ensuring interaction quality. This invention is deployed in a distributed system consisting of a video inquiry management platform, a digital human platform, and a large language model service. It can engage in dialogue with the person being questioned via video communication, and dynamically generate inquiry strategies and multi-dimensional summary reports based on the input target and background knowledge, enabling the digital human avatar to have video conversations with the person being questioned.

[0063] According to an embodiment of the present invention, the step of receiving a video query task from a business initiator through a video query management platform, and determining a video query mode based on the video query task, wherein the video query mode includes a semi-automatic video query mode and a manual video query mode, specifically:

[0064] Based on the visual query management platform, visual query instruction data from the business initiator is obtained, and a visual query task is created according to the visual query instruction data, including task identifier, visual query mode marking information, and visual query target object information;

[0065] The query mode of the business initiator is determined based on the query mode selection information of the query task.

[0066] It should be noted that the video query management platform is used to receive video query tasks initiated by business parties, create tasks, configure processes, and archive data, and has a WebRTC video channel interface.

[0067] Figure 2 A flowchart of the semi-automatic interrogation mode of the present invention is shown.

[0068] According to an embodiment of the present invention, when the interrogation mode is a semi-automatic interrogation mode, constructing a structured question list specifically involves:

[0069] The platform obtains unstructured text from the business initiator based on the video inquiry management platform. The unstructured text includes the background of the video inquiry dialogue and core objective information. The background of the video inquiry dialogue includes the type of dialogue event and the event description. The core objective information includes details of the event to be verified, points of contention, and request information.

[0070] The semantic parsing engine extracts the dialogue event type and event description from the unstructured text as the first feature vector, and extracts the details of the event to be verified and the points of contention as the second feature vector. The first and second feature vectors are input into the prompt engineering generation model. The prompt engineering generation model encodes the dialogue scene of the first feature vector to generate scene constraints through a multi-head attention mechanism, and decomposes the target intent of the second feature vector to generate target refinement conditions.

[0071] Based on the scenario constraints and target refinement conditions, a structured instruction prompt template is constructed. The structured instruction prompt template includes an event background description section, a core question generation instruction section, and an answer format requirement section.

[0072] The structured instruction prompt template is input into the instruction parsing layer of the LLM. The hierarchical decoder of the instruction parsing layer sequentially generates an event background summary, a list of key questions, and a list of optional follow-up questions. Each question item in the list of key questions is associated with a corresponding point of contention weight coefficient.

[0073] The list of key issues is prioritized based on the weight coefficients of the points of contention, and a structured list of issues is generated.

[0074] It's important to note that the original text provided by the business initiator often contains a large amount of unstructured dialogue background, event descriptions, and request information. While this information fully expresses the business needs, it lacks a systematic organizational structure and logical connections, making it difficult for the digital human system to directly recognize and ask questions. By using a semantic parsing engine to accurately extract dialogue scenario features and business objective features, and leveraging the multi-head attention mechanism of the prompting engineering generation model, the system deeply encodes the scenario and decomposes the target intent, generating structured instruction templates with clear constraints. Furthermore, by using a weighting coefficient for points of contention, the system prioritizes questions, ultimately forming a logically rigorous and focused structured question system. This not only ensures comprehensive and accurate coverage of business needs but also optimizes the questioning process through intelligent sorting. This enables the digital human to conduct efficient and systematic semi-automatic visual inquiries based on a structured question list, significantly improving interaction quality and business processing efficiency. The dialogue event type refers to the classification of the visual inquiry business scenario, such as customer complaint handling, product after-sales service, or business consultation. Scenario constraints refer to contextual limitations formed after semantic parsing and scenario encoding based on the dialogue background and event description provided by the business initiator. For example, when the dialogue event type is a customer complaint, the system will constrain the dialogue scenario to the after-sales service domain, thus limiting the LLM-generated questions to revolve around product usage, service process experience, and the occurrence of the problem, avoiding deviation from the established business scenario. Target refinement conditions, on the other hand, are targeted constraints formed after semantic decomposition of the event details and points of contention to be verified. For example, when the core objective is to verify the customer's claim of delayed delivery, target refinement conditions will guide the LLM-generated questions to focus on delivery time nodes, logistical anomalies, and the customer's expected handling methods.

[0075] According to an embodiment of the present invention, the automatic questioning operation based on the structured question list by the digital human constitutes a semi-automatic visual inquiry strategy, specifically as follows:

[0076] The digital human platform is launched and a WebRTC video session channel is established. The virtual digital human image is rendered in real time through the video session channel. When the business initiator and the interviewee successfully connect to the video session, the first question text in the current structured question list is obtained.

[0077] The question text is converted into a digital human's broadcast voice stream using text-to-speech (TTS) technology, and the question text is rendered onto the subtitle display area of ​​the user interface. The network fluctuation during the digital human's broadcast is monitored in real time, and the data packet loss rate of the current transmission link is calculated based on the network fluctuation.

[0078] When the data packet loss rate exceeds the first threshold, the encoding bitrate of the digital human video stream is reduced. Based on the time point when the data packet loss rate is greater than the preset loss rate, the broadcast voice data segment with transmission quality lower than the preset quality is determined, and the voice data segment is cached in the local buffer.

[0079] When the data packet loss rate falls below the second threshold, the untransmitted key interaction segments are extracted from the local buffer and retransmitted. The key interaction segments include the problem broadcast voice stream and the corresponding subtitle synchronization timestamp.

[0080] After the digital human finishes broadcasting the voice stream, the audio acquisition device of the interviewee is activated and the real-time speech recognition service is started. When the duration of the acquired audio of the interviewee's response exceeds the preset silence detection threshold, the audio of the response is transcribed into the response text in real time based on the speech recognition service, and the response text is updated to the response subtitle box. At the same time, the audio data packet loss rate during the transcription process is monitored.

[0081] When the audio data packet loss rate exceeds the third threshold, the current speech recognition process is interrupted and the person being questioned is prompted to repeat the answer. At the same time, the transcribed text fragments are spliced ​​together with the subsequently recorded audio of the repeated answer, and the time axis is aligned.

[0082] When the person being questioned clicks the "complete answer" command or the duration of silence in the answer audio reaches a preset termination threshold, the audio acquisition device is turned off and the current question answer text is recorded. At the same time, the semantic matching degree between the answer text and the corresponding question is calculated.

[0083] When the semantic matching degree is lower than the preset matching threshold, follow-up questions are extracted from the structured question list and inserted into the current question sequence; otherwise, the next question text is obtained according to the order of the structured question list.

[0084] The interactive loop of repeatedly executing digital human question broadcasting, data packet loss rate monitoring, audio acquisition of the queried party, lost data recovery, and semantic matching degree judgment continues until all questions in the structured question list have been interacted with, then the video session is terminated and the question and answer record text is output, thus forming a semi-automatic video query strategy.

[0085] It should be noted that in semi-automatic video interrogation mode, the digital human can automatically ask questions according to a preset structured question sequence, ensuring that no core questions are missed. Simultaneously, it improves the understanding and response accuracy of the person being questioned through both synchronized subtitles and voice broadcasting. However, in real-time video interaction based on WebRTC, network fluctuations often lead to data packet loss, resulting in issues such as voice stuttering, subtitle asynchronization, or video interruption. This invention can monitor the data packet loss rate in real time and dynamically adjust the video stream encoding bitrate based on the packet loss situation, ensuring the smoothness of video and audio. Simultaneously, it restores key audio segments and subtitle information through caching and retransmission mechanisms, avoiding the loss of important interactive content. In the answering phase, it not only supports converting the answer content into text based on Automatic Speech Recognition (ASR) technology, but also automatically interrupts and prompts the user to answer again when the audio data packet loss rate is too high. Furthermore, it uses timeline alignment technology to splice the original answer fragment with the supplementary answer into complete text, ensuring the integrity and traceability of the record. Meanwhile, this invention automatically determines whether the answer deviates from the core of the question by calculating the semantic matching degree between the answer text and the question. When the matching degree is insufficient, the system dynamically inserts follow-up questions from the structured question list to achieve targeted follow-up questions. This not only ensures the stability and integrity of audio and video interaction under complex network conditions, but also automates and intelligentizes question broadcasting and answer recognition, significantly improving the efficiency and quality of the visual inquiry process. The semantic matching degree between the answer text and the corresponding question is calculated by first converting the question text and answer text into semantic vectors using a semantic encoding model, and then calculating the semantic closeness between the two using cosine similarity to obtain a matching degree score. The recorded text includes logs of all data loss recovery operations and text fragments indicating the marked status.

[0086] Figure 3 A flowchart of the manual interrogation mode of the present invention is shown.

[0087] According to an embodiment of the present invention, when the interrogation mode is a human interrogation mode, a real-time interactive session is established, the real-time interactive session is processed for speech, and a digital human performs questioning operations based on the speech-processed interactive session to constitute a human interrogation strategy, specifically as follows:

[0088] A real-time video interactive session channel is established based on WebRTC, and the business initiator of the real-time video interactive session channel is rendered as a virtual digital human image in real time based on the digital human platform.

[0089] Real-time voice data of the business initiator and the interviewee is acquired, the real-time voice data is transcribed and the digital human voice is driven to broadcast.

[0090] When the initiator or the party being interviewed issues a command to end the interview control, the interactive record text is output, thus constituting a manual interview strategy.

[0091] It should be noted that in the human video inquiry mode, by establishing a real-time video interaction channel based on WebRTC and rendering the business initiator as a virtual digital human, the immersion and professionalism of the inquiry process are enhanced, and the tension of the inquiry recipient is avoided.

[0092] According to an embodiment of the present invention, the real-time acquisition of real-time voice data between the service initiator and the queryee, the transcription of the real-time voice data, and the driving of digital human voice playback specifically include:

[0093] The original voice streams of the service initiator and the queryee are captured based on the WebRTC real-time video interactive session channel, and the original voice streams are transmitted in real time to the voice conversion service for voiceprint feature conversion processing to generate a converted voice stream with a uniform timbre.

[0094] The converted speech stream is sent back to the digital human platform, which generates digital lip-sync control commands and speech broadcast signals to drive the virtual digital human to perform real-time speech output and lip-sync.

[0095] The automatic speech recognition engine converts the converted speech stream and the original speech stream of the interviewee into conversational interactive text in real time, and renders the conversational interactive text into the subtitle area in real time.

[0096] It should be noted that the system captures the voice streams of both parties in real time and uses a voice conversion service to process the voiceprint features, thereby generating a voice output with a uniform timbre. This avoids inconsistencies in the listening experience caused by differences in the timbre of different individuals, ensuring the neutrality and standardization of the video interview process. After the converted voice stream is sent back to the digital human platform, it can drive the digital human to achieve precise synchronization between lip movements and speech, ensuring consistency between vision and hearing and avoiding the impact of delays or misalignment on the naturalness of the interaction. At the same time, the system uses an automatic speech recognition engine to transcribe the real-time voice stream into text and establishes subtitle areas for the business initiator and the interviewee on the video interaction interface, enabling real-time subtitle presentation of the content spoken by both parties, which improves the visualization of information acquisition.

[0097] According to an embodiment of the present invention, the step of obtaining the conversation interaction text after the semi-automatic or manual interrogation strategy ends, and performing a structured transformation on the conversation interaction text based on LLM to generate a structured analysis report, specifically includes:

[0098] Obtain the conversation interaction text after the semi-automatic or manual interrogation strategy ends, and send the conversation interaction text to the LLM by calling the interrogation aggregation interface based on the digital human platform;

[0099] The conversational interaction text is structured using LLM based on advanced prompting engineering techniques to construct a structured analysis report;

[0100] The structured analysis report, conversational text, and audio / video data are compiled into a data package and sent back to the visual management platform for storage.

[0101] According to an embodiment of the present invention, the step of structuring the conversational interaction text using LLM based on advanced prompting engineering techniques to construct a structured analysis report specifically involves:

[0102] Based on LLM, the event operation flow description of the queried party in the conversation interaction text is extracted, the stage to which the event operation flow description belongs is identified, and a behavior description summary containing the event operation flow and the stage to which it belongs is generated.

[0103] Obtain standard operating procedure data for the event, compare the standard operating procedure data with the behavior description summary, and identify the event operation risk points of the queried party, including deviation from the event operation procedure and operation errors.

[0104] The behavioral description summary and event operation risk points are imported into the LLM to generate a recommendation report text, including information to be reviewed, event operation improvement suggestions, and processing time limit suggestions;

[0105] The behavioral description summary, event operation risk points, and suggestion report text are used to generate a structured analysis report.

[0106] It should be noted that by sending the conversational text to a large language model for structured processing after the semi-automatic or manual video interview, the originally unstructured dialogue content can be automatically extracted to obtain event summaries, key questions, answer points, and potential risks. This generates a logically clear and hierarchically structured analysis report, enabling efficient information organization and business value extraction. Simultaneously, packaging and encrypting the structured analysis report, original interaction text, and audio / video data before sending them back not only ensures data integrity and security but also facilitates subsequent retrieval, review, and business decision-making, thereby significantly improving the usability of the video interview results.

[0107] According to an embodiment of the present invention, it further includes:

[0108] Based on the semantic matching degree calculation result of the current question and the answer text in the semi-automatic visual query strategy, when the semantic matching degree is lower than a preset threshold, the question feature vector of the current question and the answer feature vector of the answer text are extracted.

[0109] The question feature vector and the answer feature vector are input into the context association analysis model, which calculates the question-answer feature interaction matrix through an attention mechanism to generate context association features;

[0110] Based on the contextual association features, a set of candidate follow-up questions is retrieved from the structured question list, and a correlation score between each candidate follow-up question and the current question-and-answer context is calculated.

[0111] Candidate follow-up questions are sorted according to the relevance score, and follow-up questions with relevance scores exceeding the threshold are selected to form a dynamic follow-up question sequence;

[0112] The dynamic follow-up question sequence is inserted into the original question sequence to form an updated question sequence, and the semi-automatic visual query process continues to be executed based on the updated question sequence.

[0113] According to an embodiment of the present invention, the context association analysis model calculates the question-answer feature interaction matrix through an attention mechanism to generate context association features, specifically:

[0114] A question-answer joint feature space is constructed based on the question feature vector and the answer feature vector. In the joint feature space, the cross-attention weight matrix of question features and answer features is calculated through a multi-head attention mechanism.

[0115] The question feature vector and the answer feature vector are weighted and fused according to the cross-attention weight matrix to generate context-enhanced question representation vector and answer representation vector;

[0116] The context-enhanced question representation vector and answer representation vector are concatenated and then input into a bidirectional LSTM network for temporal feature extraction, outputting contextual association features containing multi-level contextual information.

[0117] The contextual features include semantic representations in three dimensions: question intent features, answer deviation features, and potential follow-up question direction features, which are used to guide the retrieval and ranking process of subsequent follow-up questions.

[0118] It should be noted that in real-time visual inquiry digital human interaction scenarios, when the semi-automatic inquiry strategy fails to adequately match the semantics of the respondent's answer with the preset question, the system often mechanically extracts preset follow-up questions from a fixed question list, lacking a precise grasp of the deep semantic relationships within the current question-and-answer context. This leads to typical problems such as follow-up questions being out of touch with the actual situation and logical inconsistencies in the question sequence. Therefore, by constructing a contextual association analysis model based on an attention mechanism, intelligent parsing of the question-and-answer context and dynamic follow-up path planning are achieved. This method first transforms the current question and answer text into feature vector representations. In the joint feature space, a multi-head attention mechanism is used to capture the fine-grained interaction relationship between the question and the answer. The original features are dynamically weighted and fused using a cross-attention weight matrix to generate an enhanced feature representation rich in contextual semantics. Furthermore, a bidirectional LSTM network is used to extract multi-level temporal features, ultimately outputting a composite contextual association feature containing three dimensions: question intent, answer deviation direction, and potential follow-up path. By accurately identifying key information gaps or semantically ambiguous areas in responses, the system enhances the relevance and logical coherence of follow-up questions during semi-automatic interviews in practical applications. This ensures that the question sequence consistently revolves around the core points of contention, avoiding the off-topic or repetitive questioning common in traditional systems. Furthermore, by dynamically adjusting the follow-up questioning path, the system can adaptively delve deeper into key information based on the respondent's real-time responses, significantly improving the efficiency and quality of information acquisition during the interview process. This provides a more accurate and complete raw data foundation for the subsequently generated structured analysis reports, effectively addressing the adaptability issues of fixed question sequences in complex real-world situations.

[0119] Figure 4 A block diagram of a real-time visual interrogation digital human interrogation system based on WebRTC and LLM is shown.

[0120] A second aspect of the present invention also provides a real-time visual interrogation digital human interrogation system based on WebRTC and LLM. The system includes: a memory 401, a processor 402, and a communication interface 403. The memory includes a real-time visual interrogation digital human interrogation method program based on WebRTC and LLM. The communication interface is used for data connection communication between the memory and the processor. When the real-time visual interrogation digital human interrogation method program based on WebRTC and LLM is executed by the processor, it implements the steps of the real-time visual interrogation digital human interrogation method based on WebRTC and LLM as described in any of the above claims.

[0121] A third aspect of the present invention also provides a computer-readable storage medium comprising a real-time visual digital human interrogation program based on WebRTC and LLM, wherein when the real-time visual digital human interrogation program based on WebRTC and LLM is executed by a processor, it implements the steps of the real-time visual digital human interrogation method based on WebRTC and LLM as described in any of the preceding claims.

[0122] This invention discloses a real-time visual inquiry method, system, and medium based on WebRTC and LLM. The method includes: first, receiving an inquiry task that includes mode selection; second, executing differentiated processes according to the selected mode: in semi-automatic mode, an automated conversation is conducted between the digital human avatar and the inquirer based on a pre-generated structured question list from a large language model; in human mode, the digital human avatar conducts a real-time conversation with the inquirer based on real-time voice input from a human inquirer. Finally, after the conversation ends, a multi-dimensional structured summary report is automatically generated using the large language model based on the generated complete interactive text. This invention, through dual-mode design and intelligent analysis, solves the problems of low efficiency in traditional human inquiry, the awkwardness of face-to-face conversations between the inquirer and the inquirer, the negative impact on company harmony, and the time-consuming report compilation, significantly improving the standardization of inquiry.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0124] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0125] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0126] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0127] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0128] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A real-time visual interrogation method for digital humans based on WebRTC and LLM, characterized in that, Includes the following steps: The video query management platform receives video query tasks from business initiators and determines video query modes based on these tasks. The video query modes include semi-automatic video query mode and manual video query mode. When the interrogation mode is semi-automatic, a structured question list is constructed, specifically as follows: The platform obtains unstructured text from the business initiator based on the video inquiry management platform. The unstructured text includes the background of the video inquiry dialogue and core objective information. The background of the video inquiry dialogue includes the type of dialogue event and the event description. The core objective information includes details of the event to be verified, points of contention, and request information. The semantic parsing engine extracts the dialogue event type and event description from the unstructured text as the first feature vector, and extracts the details of the event to be verified and the points of contention as the second feature vector. The first and second feature vectors are input into the prompt engineering generation model. The prompt engineering generation model encodes the dialogue scene of the first feature vector to generate scene constraints through a multi-head attention mechanism, and decomposes the target intent of the second feature vector to generate target refinement conditions. Based on the scenario constraints and target refinement conditions, a structured instruction prompt template is constructed. The structured instruction prompt template includes an event background description section, a core question generation instruction section, and an answer format requirement section. The structured instruction prompt template is input into the instruction parsing layer of the LLM. The hierarchical decoder of the instruction parsing layer sequentially generates an event background summary, a list of key questions, and a list of optional follow-up questions. Each question item in the list of key questions is associated with a corresponding point of contention weight coefficient. The list of key issues is prioritized and sorted according to the weight coefficients of the points of contention, thereby generating a structured list of issues; A semi-automatic visual inquiry strategy is formed based on the digital human automatically asking questions according to the structured question list. When the video interrogation mode is the human video interrogation mode, a real-time interactive session is established, the real-time interactive session is processed by voice, and the digital human asks questions based on the voice-processed interactive session, thus forming a human video interrogation strategy. Obtain the conversation interaction text after the semi-automatic or manual interrogation strategy ends, perform structured transformation on the conversation interaction text based on LLM, and generate a structured analysis report.

2. The real-time visual interrogation digital human method based on WebRTC and LLM according to claim 1, characterized in that, The process involves receiving a video query task from the business initiator through the video query management platform, determining the video query mode based on the task, and the video query mode including a semi-automatic video query mode and a manual video query mode. Specifically: Based on the visual query management platform, visual query instruction data from the business initiator is obtained, and a visual query task is created according to the visual query instruction data, including task identifier, visual query mode marking information, and visual query target object information; The query mode of the business initiator is determined based on the query mode selection information of the query task.

3. The real-time visual interrogation digital human method based on WebRTC and LLM according to claim 1, characterized in that, The automatic questioning operation based on the structured question list by the digital human constitutes a semi-automatic visual inquiry strategy, specifically as follows: The digital human platform is launched and a WebRTC video session channel is established. The virtual digital human image is rendered in real time through the video session channel. When the business initiator and the interviewee successfully connect to the video session, the first question text in the current structured question list is obtained. The audio acquisition device of the viewed party is turned off, and the question text is converted into a digital human broadcast voice stream using text-to-speech technology. At the same time, the question text is rendered onto the subtitle display area of ​​the user interface. The network fluctuation during the digital human broadcast process is monitored in real time, and the data packet loss rate of the current transmission link is calculated based on the network fluctuation. When the data packet loss rate exceeds the first threshold, the encoding bitrate of the digital human video stream is reduced. Based on the time point when the data packet loss rate is greater than the preset loss rate, the broadcast voice data segment with transmission quality lower than the preset quality is determined, and the voice data segment is cached in the local buffer. When the data packet loss rate falls below the second threshold, the untransmitted key interaction segments are extracted from the local buffer and retransmitted. The key interaction segments include the problem broadcast voice stream and the corresponding subtitle synchronization timestamp. After the digital human finishes broadcasting the voice stream, the audio acquisition device of the interviewee is activated and the real-time speech recognition service is started. When the duration of the acquired audio of the interviewee's response exceeds the preset silence detection threshold, the audio of the response is transcribed into the response text in real time based on the speech recognition service, and the response text is updated to the response subtitle box. At the same time, the audio data packet loss rate during the transcription process is monitored. When the audio data packet loss rate exceeds the third threshold, the current speech recognition process is interrupted and the person being questioned is prompted to repeat the answer. At the same time, the transcribed text fragments are spliced ​​together with the subsequently recorded audio of the repeated answer, and the time axis is aligned. When the person being questioned clicks the "complete answer" command or the duration of silence in the answer audio reaches a preset termination threshold, the audio acquisition device is turned off and the current question answer text is recorded. At the same time, the semantic matching degree between the answer text and the corresponding question is calculated. When the semantic matching degree is lower than the preset matching threshold, follow-up questions are extracted from the structured question list and inserted into the current question sequence; otherwise, the next question text is obtained according to the order of the structured question list. The interactive loop of repeatedly executing digital human question broadcasting, data packet loss rate monitoring, audio acquisition of the queried party, lost data recovery, and semantic matching degree judgment continues until all questions in the structured question list have been interacted with, then the video session is terminated and the question and answer record text is output, thus forming a semi-automatic video query strategy.

4. The real-time visual interrogation digital human method based on WebRTC and LLM according to claim 1, characterized in that, When the interrogation mode is a human interrogation mode, a real-time interactive session is established, the real-time interactive session is processed for speech, and the digital human asks questions based on the speech-processed interactive session, thus constituting a human interrogation strategy, specifically as follows: A real-time video interactive session channel is established based on WebRTC, and the business initiator of the real-time video interactive session channel is rendered as a virtual digital human image in real time based on the digital human platform. Real-time voice data of the business initiator and the interviewee is acquired, the real-time voice data is transcribed and the digital human voice is driven to broadcast. When the initiator or the party being interviewed issues a command to end the interview control, the interactive record text is output, thus constituting a manual interview strategy.

5. The real-time visual interrogation digital human method based on WebRTC and LLM according to claim 4, characterized in that, The process of acquiring real-time voice data from the service initiator and the interviewee, transcribing the real-time voice data, and driving the digital human to broadcast the speech specifically involves: The original voice streams of the service initiator and the queryee are captured based on the WebRTC real-time video interactive session channel, and the original voice streams are transmitted in real time to the voice conversion service for voiceprint feature conversion processing to generate a converted voice stream with a uniform timbre. The converted speech stream is sent back to the digital human platform, which generates digital lip-sync control commands and speech broadcast signals to drive the virtual digital human to perform real-time speech output and lip-sync. The automatic speech recognition engine converts the converted speech stream and the original speech stream of the interviewee into conversational interactive text in real time, and renders the conversational interactive text into the subtitle area in real time.

6. The real-time visual interrogation digital human method based on WebRTC and LLM according to claim 1, characterized in that, The step involves obtaining the conversation interaction text after the semi-automatic or manual interrogation strategy has ended, and then performing a structured transformation on the conversation interaction text based on LLM to generate a structured analysis report. Specifically: Obtain the conversation interaction text after the semi-automatic or manual interrogation strategy ends, and send the conversation interaction text to the LLM by calling the interrogation aggregation interface based on the digital human platform; The conversational interaction text is structured using LLM based on advanced prompting engineering techniques to construct a structured analysis report; The structured analysis report, conversational text, and audio / video data are compiled into a data package and sent back to the visual management platform for storage.

7. The real-time visual interrogation digital human method based on WebRTC and LLM according to claim 6, characterized in that, The step of using LLM to structure the conversational interaction text based on advanced prompting engineering techniques and constructing a structured analysis report specifically involves: Based on LLM, the event operation flow description of the queried party in the conversation interaction text is extracted, the stage to which the event operation flow description belongs is identified, and a behavior description summary containing the event operation flow and the stage to which it belongs is generated. Obtain standard operating procedure data for the event, compare the standard operating procedure data with the behavior description summary, and identify the event operation risk points of the queried party, including deviation from the event operation procedure and operation errors. The behavioral description summary and event operation risk points are imported into the LLM to generate a recommendation report text, including information to be reviewed, event operation improvement suggestions, and processing time limit suggestions; The behavioral description summary, event operation risk points, and suggestion report text are used to generate a structured analysis report.

8. A real-time visual interrogation digital human interrogation system based on WebRTC and LLM, characterized in that, The WebRTC and LLM-based real-time visual digital human interrogation system includes a storage unit and a processor. The storage unit includes a WebRTC and LLM-based real-time visual digital human interrogation method program. When the WebRTC and LLM-based real-time visual digital human interrogation method program is executed by the processor, it implements the steps of the WebRTC and LLM-based real-time visual digital human interrogation method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a real-time visual digital human interrogation program based on WebRTC and LLM. When the real-time visual digital human interrogation program based on WebRTC and LLM is executed by a processor, it implements the steps of the real-time visual digital human interrogation method based on WebRTC and LLM as described in any one of claims 1 to 7.

Images