Virtual human interaction control method and device for cross-modal identity authentication and medium

Abstract

The invention discloses a virtual human interaction control method and device for cross-modal identity authentication and a medium, and relates to the technical field. The method comprises the following steps: collecting user face data through a visual sensor, and collecting user voice data through an audio sensor; based on the face data and the voice data, calculating a user identity posteriori through a multi-modal identity recognition module, calculating an emotion intensity value through a multi-modal emotion recognition module, and converting the user voice data into a user question text through a voice recognition module; based on the user identity posteriori, the emotion intensity value and the user question text, generating an answer text through a large model; based on the emotion intensity value, selecting at least one action from a preset action library through an action control module; and the answer text is converted into voice output through a voice synthesis module, and the selected action and the voice output synchronously drive the virtual human to perform expression. According to the method, intelligent and personified improvement of virtual human interaction control is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field, and particularly relates to a virtual human interaction control method and device based on cross-modal identity identification and a medium. BACKGROUND

[0002] With the development of artificial intelligence technology, digital humans have been widely applied to interactive scenarios such as customer service, virtual tour, online education, etc. The existing digital human interaction system usually relies on a pre-set dialogue process or text content generated based on a large language model for voice broadcasting, and to some extent, combines simple expressions or actions.

[0003] Such systems still have obvious deficiencies in realizing natural and personified interactive experience. First, the recognition method of most systems is single, usually only relying on one of face recognition or voiceprint recognition, and under real conditions such as poor lighting, face occlusion or noisy environment, the recognition stability significantly decreases, leading to interruption of interaction or misjudgment of identity. Second, the real-time emotional state of the user is difficult to be effectively perceived and utilized by the system, even if some systems attempt to perform emotion recognition, they are often independent of subsequent semantic generation, speech synthesis and action control, leading to a lack of consistency between the language content, voice tone, facial expression and body movement of the digital human, and the overall performance is stiff and mechanical, and it is difficult to establish emotional resonance.

[0004] Through the above analysis, the problems and defects of the prior art are: The digital human interaction system in the prior art is unstable in the interaction process, has low personification degree and lacks emotional consistency due to single identity recognition mode, disconnection between emotion perception and semantic and action control. SUMMARY

[0005] The embodiment of the present application provides a virtual human interaction control method and device based on cross-modal identity identification and a medium, which can solve the problems of unstable interaction process, low personification degree and lack of emotional consistency of the digital human interaction system in the prior art due to single identity recognition mode, disconnection between emotion perception and semantic and action control.

[0006] In a first aspect, the embodiments of the present application provide a virtual human interaction control method for cross-modal identity authentication, which comprises: collecting user facial data through a visual sensor and collecting user voice data through an audio sensor; calculating a user identity posterior through a multi-modal identity recognition module based on the facial data and the voice data, calculating an emotion intensity value through a multi-modal emotion recognition module, and converting the user voice data into user question text through a voice recognition module; generating an answer text based on the user identity posterior, the emotion intensity value and the user question text through a large model; selecting at least one action from a preset action library based on the emotion intensity value, wherein the actions in the preset action library are associated with emotion intensity value intervals; and converting the answer text into voice output through a voice synthesis module, and driving a virtual human to perform the selected action and the voice output synchronously.

[0007] In an implementation manner of the present application, the user identity posterior is calculated through the multi-modal identity recognition module, specifically comprising: extracting a facial feature vector and a voiceprint feature vector of a current frame based on the facial data and the voice data; calculating a similarity of the facial feature vector and a pre-stored facial template vector, and a similarity of the voiceprint feature vector and a pre-stored voiceprint template vector; adjusting a visual modal weight and an audio modal weight according to a visual modal confidence and an audio modal confidence; performing weighted summation on the similarities using the adjusted weights to obtain a fusion similarity; and inputting the fusion similarity into a pre-trained vector model to retrieve and output the user identity posterior.

[0008] In an implementation manner of the present application, the emotion intensity value is calculated through the multi-modal emotion recognition module, specifically comprising: outputting a visual emotion likelihood estimate based on the facial data, and outputting an audio emotion likelihood estimate based on the voice data; adjusting a visual modal weight and an audio modal weight according to a visual modal confidence and an audio modal confidence; performing weighting on the visual emotion likelihood estimate and the audio emotion likelihood estimate respectively, and calculating an emotion posterior distribution in combination with a prior emotion distribution; performing normalization processing on the emotion posterior distribution; and performing weighted averaging on an emotion intensity value corresponding to the visual emotion likelihood estimate and an emotion intensity value corresponding to the audio emotion likelihood estimate to obtain a normalized emotion intensity value.

[0009] In an implementation manner of the present application, the answer text is generated through the large model based on the user identity posterior, the emotion intensity value and the user question text, specifically comprising: retrieving corresponding user identity information in a user information library according to the user identity posterior, the user information library at least storing a mapping relationship between a user identity identifier and a corresponding title; determining an emotion regulation strategy according to the emotion intensity value, the emotion regulation strategy including inserting soothing phrases, exclamation words or adjusting the formal degree of a sentence in the answer text that match the emotion intensity.

[0010] In an implementation form of the present application, the method further comprises: constructing a large model input prompt word, the prompt word integrating the user question text, the user identity information, the emotion regulation strategy and the instruction of generating the answer; inputting the prompt word into the large model to generate a preliminary answer text; performing compliance checking on the preliminary answer text, and taking the answer text passing the compliance checking as the final answer text.

[0011] In an implementation form of the present application, based on the emotion intensity value, at least one action is selected from a preset action library by an action control module, specifically comprising: reading the emotion intensity value; traversing the preset action library, wherein each action is associated with a trigger lower limit and a trigger upper limit; selecting actions whose emotion intensity values fall between the trigger lower limit and the trigger upper limit; if multiple actions are selected, a single action is selected by a random sampling rule.

[0012] In an implementation form of the present application, the method further comprises: prompting the user for data collection type, purpose and storage period, and obtaining user confirmation; encrypting and storing the collected facial data and voice data; discarding the facial data and voice data after converting them into facial feature vectors and voiceprint feature vectors.

[0013] In an implementation form of the present application, the method further comprises: extracting the rhythm and accent time points of the voice based on the voice waveform output by the voice synthesis module; based on the rhythm and accent time points, generating a time axis of action execution, so that the key actions are aligned with the accent or pause moments in the voice.

[0014] In a second aspect, the embodiments of the present application also provide a virtual human interaction control device for cross-modal identity identification, the device comprising at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: collect user facial data through a visual sensor and collect user voice data through an audio sensor; calculate a user identity posteriori through a multi-modal identity recognition module and calculate an emotion intensity value through a multi-modal emotion recognition module based on the facial data and the voice data; convert the user voice data into a user question text through a voice recognition module; generate an answer text based on the user identity posteriori, the emotion intensity value and the user question text through a large model; select at least one action from a preset action library through an action control module based on the emotion intensity value, wherein the actions in the preset action library are associated with emotion intensity value intervals; convert the answer text into a voice output through a voice synthesis module, and drive a virtual human to perform the selected action and the voice output synchronously.

[0015] In a third aspect, the embodiments of the present application also provide a non-volatile computer storage medium for cross-modal identity authentication virtual human interaction control, which stores computer executable instructions. The computer executable instructions are configured to: collect user facial data through a visual sensor and collect user voice data through an audio sensor; calculate a user identity posterior through a multi-modal identity recognition module, calculate an emotional intensity value through a multi-modal emotion recognition module, and convert the user voice data into user question text through a voice recognition module based on the facial data and the voice data; generate an answer text based on the user identity posterior, the emotional intensity value, and the user question text through a large model; select at least one action from a preset action library based on the emotional intensity value through an action control module, wherein the actions in the preset action library are associated with an emotional intensity value interval; and convert the answer text into voice output through a voice synthesis module, and drive a virtual human to perform through the selected action and the voice output.

[0016] The embodiments of the present application provide a cross-modal identity authentication virtual human interaction control method, device and medium. Through the fusion and weight self-adaptive adjustment of visual and audio modalities, the accuracy and robustness of identity and emotion recognition are significantly improved, and stable interaction in a complex real environment is ensured. The recognized user identity and real-time emotional intensity value are deeply integrated into the answer generation logic of the large model, so that the reply content has personalization and emotional adaptability, greatly enhancing the affinity and pertinence of the interaction. The selection of the preset action library is innovatively driven by the quantitative emotional intensity value, and is accurately synchronized with the rhythm and stress of the voice synthesis output, realizing multi-level and consistent expression of digital human semantics, voice and action, and creating a highly natural personification interaction experience. Through the whole-process privacy control mechanism, the safety and compliance of user sensitive data are ensured while the function implementation is guaranteed, and the credibility of the system is improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the illustrative embodiments of the present application and its description, and do not constitute improper limitations on the present application. In the drawings: Figure 1 A flowchart of a cross-modal identity authentication virtual human interaction control method provided by the embodiments of the present application; Figure 2 An internal structure schematic diagram of a cross-modal identity authentication virtual human interaction control device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0018] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0019] The embodiment of the present application provides a virtual human interaction control method for cross-modal identity identification, device and medium, which solves the problems of unstable interaction process, low personification degree and lack of emotional consistency caused by single identity identification mode, disconnection between emotion perception and semantic and action control in the digital human interaction system in the prior art.

[0020] The technical solutions of the embodiments of the present application will be described in detail below with reference to the drawings.

[0021] Figure 1 A flow chart of a virtual human interaction control method for cross-modal identity identification is provided in the embodiment of the present application. As shown in Figure 1 The virtual human interaction control method for cross-modal identity identification provided in the embodiment of the present application specifically includes the following steps: Step 10: collecting user facial data through a visual sensor and collecting user voice data through an audio sensor; Step 20: calculating user identity posteriori through a multi-modal identity identification module, calculating emotion intensity value through a multi-modal emotion identification module, and converting user voice data into user question text through a voice identification module based on facial data and voice data; In the actual digital human interaction scene, a single mode is often insufficient to support stable identity and emotion identification in this step.

[0022] As an optional embodiment, calculating user identity posteriori through a multi-modal identity identification module can specifically include: step 2011: extracting facial feature vectors and voiceprint feature vectors of the current frame based on facial data and voice data; step 2012: calculating the similarity of the facial feature vectors and the pre-stored facial template vectors, and the similarity of the voiceprint feature vectors and the pre-stored voiceprint template vectors; step 2013: adjusting the visual modal weight and the audio modal weight according to the visual modal confidence and the audio modal confidence; step 2014: using the adjusted weights to weight and sum the similarities to obtain a fusion similarity; step 2015: inputting the fusion similarity into a pre-trained vector model to retrieve and output user identity posteriori.

[0023] In this step, for example, when the camera picture is insufficiently illuminated, the user's face is partially obscured, the visual-based face feature vector will be affected. Similarly, when the environmental noise is too large or the user's voice is loud, the voiceprint-related feature variable will also be greatly affected. To solve this problem, the present application adopts a multi-modal evidence aggregation strategy to fuse the visual and audio judgments into a unified posteriori, and inputs the unified posteriori into a vector model trained by pre-collected voice and video of the person to be identified for retrieving the identity of the person.

[0024] Identity posteriori fusion:

[0025]

[0026] : current frame face vector; : face center vector of person ii, collected by the camera and processed into a vector by the embedding model; , : voiceprint vector and template, collected by the microphone and generated by the embedding model; : cosine similarity (or dot product); : modal weight (determined by modal confidence / environmental adaptation); : temperature parameter; : search using a pre-trained vector model, retrieve the person id; : matched person identity.

[0027] If the camera is obscured, assuming the user is in a dimly lit room, the camera picture is blurred, the system will automatically reduce and increase , so that the audio dominates the identity determination.

[0028] As an optional embodiment, the emotion intensity value is calculated by a multi-modal emotion recognition module, which can specifically include: step 2021: output a visual emotion likelihood estimate based on face data, and output an audio emotion likelihood estimate based on voice data; step 2022: adjust the visual modal weight and the audio modal weight according to the visual modal confidence and the audio modal confidence; step 2023: weight the visual emotion likelihood estimate and the audio emotion likelihood estimate respectively, and calculate an emotion posteriori distribution in combination with a priori emotion distribution; step 2024: normalize the emotion posteriori distribution; step 2025: weight average the emotion intensity value corresponding to the visual emotion likelihood estimate and the emotion intensity value corresponding to the audio emotion likelihood estimate, to obtain a normalized emotion intensity value.

[0029] In this step, in the digital human interaction system, the emotion recognition result of a single modality often has instability. Therefore, the system fuses the results of the visual model and the audio model to form a more robust emotion posterior distribution.

[0030]

[0031] After normalization, Pt(e) is obtained, and the intensity is fused:

[0032] The likelihood estimate of the visual / audio model for the emotion category; The weight is the same as above; The fused emotion intensity (0-1). Used to determine whether to calm down or slow down the speech speed, and as an input to the subsequent action generation module.

[0033] Step 30: Based on the user identity posterior, the emotion intensity value, and the user question text, generate an answer text through a large model; As an optional embodiment, based on the user identity posterior, the emotion intensity value, and the user question text, generate an answer text through a large model, which can specifically include: step 301: according to the user identity posterior, retrieve the corresponding user identity information in the user information library, the user information library at least stores the mapping relationship between the user identity identifier and the corresponding title; step 302: according to the emotion intensity value, determine the emotion regulation strategy, which includes inserting soothing language, exclamation words or adjusting the formality of the answer text that matches the emotion intensity.

[0034] In this step, multiple factors such as multi-modal evidence, emotion posterior fusion results, and user identity information need to be considered to ensure that the final generated reply text not only has reasonable semantics, but also meets the requirements of politeness and individualization.

[0035] Specifically, first, the user's voice input is collected, and the transcription result is obtained through a speech recognition model as the user question At the same time, the multi-modal identity recognition module will infer the user's identity based on the user's historical information and the current interaction context And the emotion posterior module gives the normalized emotion intensity at the current time based on multi-modal evidence, such as speech acoustic features, semantic expression, facial expression, etc.

[0036] On this basis, the large model needs to generate the final reply text And combined with controllable parameters such as title, politeness level, soothing language insertion, etc., to ensure that the text output not only has a smooth logic, but also meets the business constraints and user experience requirements. ​

[0037] The generated large model answer will be played through the microphone.

[0038]

[0039] : Call large model for processing; : User identity output by multi-modal identity recognition module; : Recognized user question; : Emotional intensity obtained by emotion recognition module.

[0040] As an optional embodiment, the method can further include: step 303: constructing a large model input prompt word, the prompt word integrating user question text, user identity information, emotional regulation strategy and generation answer instruction; step 304: inputting the prompt word into the large model to generate a preliminary answer text; step 305: performing compliance check on the preliminary answer text, and taking the answer text passing the compliance check as the final answer text.

[0041] Step 40: based on the emotional intensity value, selecting at least one action from the preset action library through the action control module, wherein the actions in the preset action library are associated with the emotional intensity value interval; As an optional embodiment, based on the emotional intensity value, selecting at least one action from the preset action library through the action control module, specifically can include: step 401: reading the emotional intensity value; step 402: traversing the preset action library, wherein each action is associated with a trigger lower limit and a trigger upper limit; step 403: selecting actions whose emotional intensity values fall between the trigger lower limit and the trigger upper limit; step 404: if multiple actions are selected, selecting a single action through a random sampling rule.

[0042] In this step, according to the emotional intensity value output by the emotion recognition module , one or more appropriate response actions are selected from a series of preset action libraries, so that the intelligent agent can naturally interact with the user's emotional state through action feedback.

[0043] Input: emotional intensity value , given by the emotion recognition module.

[0044] Action library: , wherein each action has a pre-defined threshold interval triggered.

[0045] Output: action set conforming to the current emotional intensity interval , if only a single action needs to be output, a priority or random sampling rule can be defined.

[0046] Each action The trigger condition of each action can be expressed as:

[0047] Wherein: The trigger lower bound and upper bound of each action When , it means that the emotional intensity falls into the threshold interval of the action, and the action is selected.

[0048] The final output action set is:

[0049] Table 1 Example action library

[0050] Read the emotional intensity from the emotion recognition module ; traverse the action configuration table to find all actions that meet ; if multiple actions are matched at the same time, use random sampling to enhance diversity using the following temperature sampling parameters. Pass the selected action to the action control module, such as the 3D digital human, robot action engine, to complete the action performance.

[0051] Step 50: Convert the answer text to speech output through the speech synthesis module, synchronize the selected action with the speech output, and drive the virtual person to perform.

[0052] As an optional embodiment, the method can further include: prompting the user for data collection type, purpose, and storage period, and obtaining user confirmation; encrypting and storing the collected facial data and voice data; converting the facial data and voice data into facial feature vectors and voiceprint feature vectors, and then discarding them.

[0053] In this step, in the digital human interaction system, it involves the collection of multi-modal data such as camera facial information and microphone voice information, which belong to sensitive personal privacy data. In order to ensure the compliance of the system and the trust of the user, this module introduces a privacy control and compliance mechanism to protect the data in the whole process of collection, storage, processing and invocation.

[0054] Before data collection starts, the system will prompt the user through the digital human or interface, such as explaining the type of collection, video or audio, the purpose of collection, identity recognition or emotion analysis, storage period, etc. For the original collected audio and video data, AES-256 symmetric encryption method is used for storage. The encryption formula is as follows:

[0055] : original data block; : system-generated symmetric key, stored only in secure hardware module; : encryption function; : encrypted ciphertext.

[0056] Before the data is transmitted into the multi-modal identity recognition and emotion recognition module, the audio and video signals are converted into embedding feature vectors , and the original data frames are discarded. In this way, it can be ensured that the system only processes non-reversible feature vectors during inference, reducing privacy risks.

[0057] All access to encrypted data must be authenticated, and the system will record access logs, including calling modules, time, purpose, to ensure traceability.

[0058] As an optional embodiment, the method can further include: extracting the rhythm and accent time points of the voice according to the voice waveform output by the voice synthesis module; and generating a time axis of action execution based on the rhythm and accent time points, so that the key actions are aligned with the accent or pause moments in the voice.

[0059] The above is the method embodiment of the present application. Based on the same inventive concept, the present application also provides a virtual human interaction control device for cross-modal identity identification, the structure of which is shown in Figure 2 .

[0060] Figure 2 A virtual human interaction control device for cross-modal identity identification provided by an embodiment of the present application is shown in Figure 2 . The device includes: at least one processor 201; and a memory 202 in communication connection with the at least one processor; Wherein the memory 202 stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor 201 to enable the at least one processor 201 to: collect user facial data through a visual sensor and collect user voice data through an audio sensor; based on the facial data and the voice data, calculate the user identity posterior through a multi-modal identity recognition module, calculate the emotion intensity value through a multi-modal emotion recognition module, and convert the user voice data into user question text through a voice recognition module; based on the user identity posterior, the emotion intensity value and the user question text, generate an answer text through a large model; based on the emotion intensity value, select at least one action from a preset action library through an action control module, wherein the actions in the preset action library are associated with the emotion intensity value interval; convert the answer text into voice output through a voice synthesis module, and drive a virtual human to perform the selected action and the voice output synchronously.

[0061] Some embodiments of the present application provide a non-transitory computer storage medium storing computer executable instructions of a cross-modal identity authentication virtual human interaction control, corresponding to Figure 1 The computer executable instructions are configured to: collect user facial data through a visual sensor, and collect user voice data through an audio sensor; calculate a user identity posterior through a multi-modal identity recognition module, calculate an emotion intensity value through a multi-modal emotion recognition module, and convert the user voice data into user question text through a voice recognition module based on the facial data and the voice data; generate an answer text based on the user identity posterior, the emotion intensity value, and the user question text through a large model; select at least one action from a preset action library through an action control module based on the emotion intensity value, wherein the actions in the preset action library are associated with emotion intensity value intervals; and drive a virtual human to perform through a voice synthesis module converting the answer text into voice output and synchronously driving the selected action and the voice output.

[0062] Each of the embodiments in the present application is described in a progressive manner, and the same or similar parts of each of the embodiments can be referred to each other. Each of the embodiments focuses on the differences from other embodiments. In particular, the IoT device and medium embodiments are basically similar to the method embodiments, and thus are described simply. The relevant parts can be referred to the part of the method embodiments.

[0063] The system and medium provided by the embodiments of the present application are one-to-one corresponding to the method. Therefore, the system and medium also have similar beneficial technical effects as the method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the system and medium will not be described here.

[0064] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0065] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0066] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0067] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0068] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0069] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, for storing instructions and data used and / or generated by the computing device. The memory can also include non-volatile memory, such as read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or nonvolatile random access memory (NVRAM) for storing instructions and data used and / or generated by the computing device. The memory is an example of computer readable media.

[0070] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0071] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0072] The above only describes the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A virtual human interaction control method for cross-modal identity authentication, characterized in that, The method includes: The system collects user facial data using a visual sensor and user voice data using an audio sensor. Based on the facial and voice data, the user's identity posterior is calculated by the multimodal identity recognition module, the emotion intensity value is calculated by the multimodal emotion recognition module, and the user's voice data is converted into user question text by the voice recognition module. Based on the user identity posterior, emotion intensity value, and user question text, a large model is used to generate the response text. Based on the emotion intensity value, at least one action is selected from a preset action library by the action control module, wherein the actions in the preset action library are associated with the emotion intensity value range; The speech synthesis module converts the response text into speech output, and the selected action is synchronized with the speech output to drive the virtual human to perform.

2. The virtual human interaction control method for cross-modal identity authentication according to claim 1, characterized in that, The calculation of the user identity posterior through the multimodal identity recognition module specifically includes: Based on the facial data and voice data, extract the facial feature vector and voiceprint feature vector of the current frame; Calculate the similarity between the facial feature vector and the pre-stored facial template vector, and the similarity between the voiceprint feature vector and the pre-stored voiceprint template vector; The visual modality weights and audio modality weights are adjusted based on the visual modality confidence and audio modality confidence. The similarities are weighted and summed using the adjusted weights to obtain the fused similarity. The fused similarity is input into the pre-trained vector model to retrieve and output the user identity posterior.

3. The virtual human interaction control method for cross-modal identity authentication according to claim 1, characterized in that, The calculation of emotion intensity value through the multimodal emotion recognition module specifically includes: Based on the facial data, output a visual emotion likelihood estimate, and based on the speech data, output an audio emotion likelihood estimate; The visual modality weights and audio modality weights are adjusted based on the visual modality confidence and audio modality confidence. The visual emotion likelihood estimate and the audio emotion likelihood estimate are weighted separately, and combined with the prior emotion distribution, the posterior emotion distribution is calculated. The posterior distribution of the emotion is normalized. The normalized emotion intensity value is obtained by taking a weighted average of the emotion intensity values ​​corresponding to the visual emotion likelihood estimation and the audio emotion likelihood estimation.

4. The virtual human interaction control method for cross-modal identity authentication according to claim 1, characterized in that, The process of generating response text based on the user's identity posterior, emotion intensity value, and user question text using a large model specifically includes: Based on the user identity verification, the corresponding user identity information is retrieved from the user information database, which at least stores the mapping relationship between user identity identifiers and corresponding titles; Based on the emotional intensity value, an emotion regulation strategy is determined, which includes inserting reassuring words, interjections, or adjusting the formality of statements in the response text to match the emotional intensity.

5. The virtual human interaction control method for cross-modal identity authentication according to claim 4, characterized in that, The method further includes: Construct a large model input prompt word, which integrates the user question text, user identity information, sentiment regulation strategy and instructions for generating an answer; Input the prompt words into the large model to generate a preliminary response text; The preliminary response text is subjected to a compliance check, and the response text that passes the compliance check is used as the final response text.

6. The virtual human interaction control method for cross-modal identity authentication according to claim 1, characterized in that, The step of selecting at least one action from a preset action library based on the emotion intensity value via the action control module specifically includes: Read the emotional intensity value; Iterate through the pre-defined action library, where each action is associated with a lower trigger bound and an upper trigger bound; Select all actions whose emotional intensity values ​​fall between the lower and upper trigger bounds. If multiple actions are selected, a single action is selected based on random sampling rules.

7. The virtual human interaction control method for cross-modal identity authentication according to claim 1, characterized in that, The method further includes: Inform the user of the data collection type, purpose, and storage period, and obtain the user's confirmation; The collected facial and voice data are encrypted and stored. The facial and voice data are converted into facial feature vectors and voiceprint feature vectors and then discarded.

8. The virtual human interaction control method for cross-modal identity authentication according to claim 1, characterized in that, The method further includes: Based on the speech waveform output by the speech synthesis module, extract the rhythm and stress points of the speech. Based on the rhythm and stress points, a timeline for action execution is generated, aligning key actions with stress points or pauses in the speech.

9. A virtual human interaction control device for cross-modal identity authentication, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: The system collects user facial data using a visual sensor and user voice data using an audio sensor. Based on the facial and voice data, the user's identity posterior is calculated by the multimodal identity recognition module, the emotion intensity value is calculated by the multimodal emotion recognition module, and the user's voice data is converted into user question text by the voice recognition module. Based on the user identity posterior, emotion intensity value, and user question text, a large model is used to generate the response text. Based on the emotion intensity value, at least one action is selected from a preset action library by the action control module, wherein the actions in the preset action library are associated with the emotion intensity value range; The speech synthesis module converts the response text into speech output, and the selected action is synchronized with the speech output to drive the virtual human to perform.

10. A non-volatile computer storage medium for cross-modal identity authentication and virtual human interaction control, storing computer-executable instructions, characterized in that, The computer-executable instructions are set as follows: The system collects user facial data using a visual sensor and user voice data using an audio sensor. Based on the facial and voice data, the user's identity posterior is calculated by the multimodal identity recognition module, the emotion intensity value is calculated by the multimodal emotion recognition module, and the user's voice data is converted into user question text by the voice recognition module. Based on the user identity posterior, emotion intensity value, and user question text, a large model is used to generate the response text. Based on the emotion intensity value, at least one action is selected from a preset action library by the action control module, wherein the actions in the preset action library are associated with the emotion intensity value range; The speech synthesis module converts the response text into speech output, and the selected action is synchronized with the speech output to drive the virtual human to perform.

Images