A virtual digital human interaction management method and system based on artificial intelligence

Abstract

The present invention discloses a virtual digital human interaction management method and system based on artificial intelligence, which relates to the field of artificial intelligence technology, including: structurally decomposing a task requirement description, and assigning the decomposed subtasks to digital humans with corresponding capabilities; each digital human retrieves historical context data and executes the task according to the received subtask, and generates a subtask execution result; the priority and output form of the task execution result are adjusted according to the user's emotional state, and an integrated collaborative output result is generated; based on the collaborative output result, the main digital human analyzes the user's real-time emotional fluctuations and dynamically adjusts the main digital human's output mode. By analyzing the user's emotional fluctuation data in real time, the interactive tone, output form and content complexity of the main digital human are dynamically adjusted, thereby achieving a more humanized interactive mode.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based virtual digital human interaction management method and system. Background Art

[0002] With the rapid development of artificial intelligence technology, virtual digital humans, as an emerging intelligent interactive subject, have been widely used in intelligent customer service, education and training, virtual assistants and other fields. Through the collection and understanding of multimodal data (such as voice, expression, gesture), virtual digital humans can achieve natural interaction with users and meet the needs of diverse scenarios. Current virtual digital human interaction systems usually use voice recognition, text analysis or simple emotion detection models to obtain user intentions, but these single-modal interaction methods are unable to cope with complex user emotions and dynamic needs. In addition, existing technologies mostly focus on static task allocation and execution, lack the ability to adjust in real time to user emotional fluctuations, and it is difficult to achieve personalized and emotional interactive experience.

[0003] However, the existing virtual digital human interaction technology has the following shortcomings: on the one hand, the fusion processing ability of multimodal data is weak, and it is unable to accurately analyze the complex intentions and emotional states of users, resulting in insufficient naturalness and accuracy of the interaction; on the other hand, in the process of task decomposition and execution, there is a lack of dynamic perception and response mechanism for user emotions. The existing system usually presents task results in a fixed output form and fails to adjust the interaction method according to user emotions. This not only reduces the user experience, but also may cause negative effects in some sensitive scenarios (such as medical consultation or psychological counseling). Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a virtual digital human interaction management method based on artificial intelligence to solve the problems of insufficient multimodal data fusion processing and lack of emotional dynamic perception response in existing virtual digital human interaction technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a virtual digital human interaction management method based on artificial intelligence, which includes collecting multimodal data of users and performing standardized processing;

[0008] Through natural language processing technology, the standardized multimodal data is analyzed to generate task requirement descriptions of user intentions and emotional states;

[0009] Decompose the task requirement description into a structured manner and assign the decomposed subtasks to digital humans with corresponding capabilities;

[0010] Each digital human retrieves historical context data and executes tasks according to the received subtasks, generating subtask execution results;

[0011] Adjust the priority and output form of task execution results according to the user's emotional state to generate integrated collaborative output results;

[0012] Based on the collaborative output results, the master digital human analyzes the user's real-time emotional fluctuations and dynamically adjusts the master digital human's output method.

[0013] As a preferred solution of the virtual digital human interaction management method based on artificial intelligence of the present invention, wherein: the multimodal data includes the user's voice signal, facial expression data and gesture trajectory data;

[0014] The standardization process comprises the following steps:

[0015] Perform noise reduction and feature extraction on the user's voice signal to generate a voice feature vector;

[0016] Normalize and enhance the facial expression data to generate facial expression feature vectors;

[0017] The gesture trajectory data is smoothed and time series sampled to generate the gesture trajectory feature vector.

[0018] As a preferred solution of the virtual digital human interaction management method based on artificial intelligence described in the present invention, wherein: parsing the standardized multimodal data through natural language processing technology to generate a task requirement description of the user's intention and emotional state includes the following steps:

[0019] Use the speech recognition model to perform speech-to-text processing on the speech feature vector to generate a text representation of the user's speech;

[0020] Perform word segmentation on the generated text, remove stop words and perform lemma restoration to obtain a word sequence;

[0021] Use the pre-trained natural language understanding model to semantically vectorize the word sequence and generate the semantic embedding vector of the text;

[0022] Use the classification model to classify the semantic embedding vector of the text and identify the user's main intention;

[0023] The facial expression feature vector is classified by convolutional neural network to obtain the user's label emotion category;

[0024] The gesture trajectory feature vector is input into the long short-term memory network as a time series to extract the dynamic emotional features of the gesture and obtain the user's gesture emotion category;

[0025] The user's label emotion and gesture emotion are integrated to obtain the final emotional state;

[0026] The semantic embedding vector and the final emotional state are fused to generate a comprehensive feature vector;

[0027] The task generation model is used to reason over the comprehensive feature vector to generate a structured task requirement description.

[0028] As a preferred solution of the virtual digital human interaction management method based on artificial intelligence described in the present invention, wherein: structurally decomposing the task requirement description and assigning the decomposed subtasks to digital humans with corresponding capabilities includes the following steps:

[0029] Decompose the text part of the task requirement description through dependency parsing to extract the main verbs of the task;

[0030] Structural modeling of tasks is performed based on the main verbs of the tasks to generate a task structure diagram;

[0031] The emotional state and context information in the task requirement description are embedded into the task structure diagram to obtain the final task structure diagram;

[0032] Establish capability vectors for each digital human in the collaborative network;

[0033] For each task node in the final task structure diagram, calculate its matching degree with each digital human capability vector, and the expression is:

[0034] ;

[0035] in, Represents a task node Human Ability Vector with Numbers The degree of adaptation, Indicates Task nodes, Indicates the task node index, represents the capability vector of the digital human, Indicates that the task node is The importance of the field An index representing the task domain, represents the number of task areas, Indicates that digital people are in Ability value in each field, represents the adjustment factor, Indicates The complexity of requirements in each area;

[0036] According to the matching degree, the task nodes are assigned to the digital human with the highest matching degree, and a task allocation table is generated;

[0037] Send the subtasks in the task assignment table to digital humans with corresponding capabilities, and build a collaborative mechanism to coordinate the order of data transmission between digital humans.

[0038] As a preferred solution of the method for managing virtual digital human interaction based on artificial intelligence described in the present invention, each digital human retrieves historical context data and executes tasks according to the received subtasks, and generating subtask execution results includes the following steps:

[0039] The digital human extracts contextual data related to the subtask from the task requirement description and retrieves it through the knowledge graph;

[0040] Based on the retrieved context data, construct API query parameters to obtain real-time data;

[0041] Analyze the nodes related to the subtask in the knowledge graph and perform node reasoning through the graph neural network. The expression is:

[0042] ;

[0043] in, Representation Node in the The node feature representation of the layer, It represents the activation function that adds nonlinear characteristics to the result after linear transformation. Indicates The weight matrix of the layer, represents neighbor nodes, Representation Node The set of neighbor nodes of Representation and Node Connected neighbor nodes In the The feature representation of the layer, Represents the graph neural network in The bias vector of the layer, represents a node in the graph, Indicates the current number of layers in the graph neural network;

[0044] According to the task requirements, select the nodes related to the current reasoning task target, and filter out the user demand features through the query conditions in the knowledge graph, and use the node feature representation to score. The expression is:

[0045] ;

[0046] in, Representation Node The relevance score to the task user demand characteristics, Indicates the characteristics of user needs. represents the context weight, Represents contextual features;

[0047] Sort the nodes based on the relevance scores and generate recommendations based on the sorting results;

[0048] Digital people adjust strategies based on recommended content and screen out qualified solutions;

[0049] Generate personalized subtask execution results based on the screened qualified solutions.

[0050] As a preferred solution of the virtual digital human interaction management method based on artificial intelligence described in the present invention, wherein: adjusting the priority and output form of the task execution result according to the user's emotional state, generating the integrated collaborative output result includes the following steps:

[0051] Integrate the sub-task execution results of each digital human into a preliminary structured data set;

[0052] According to the user's current emotional state, the priority of each task result in the structured data set is adjusted, and sorted based on the priority, the content and presentation form of the results are adjusted, and the integrated collaborative output results are generated and delivered to the main digital human.

[0053] As a preferred solution of the method for managing virtual digital human interaction based on artificial intelligence described in the present invention, wherein: based on the collaborative output result, the main digital human analyzes the real-time emotional fluctuations of the user, and dynamically adjusts the output mode of the main digital human, including the following steps:

[0054] The master digital human receives the integrated collaborative output results from the collaborative network and compares them with the multimodal data collected in real time to detect the user's emotional fluctuations;

[0055] Update the current emotional state according to the detection results, and adjust the interactive tone, output form and complexity of the output content of the main digital human;

[0056] Based on the adjusted tone, output form and output content, the master digital human generates the final output content.

[0057] In a second aspect, the present invention provides a virtual digital human interaction management system based on artificial intelligence, including a data acquisition module, which collects multimodal data of users and performs standardized processing;

[0058] The intention analysis module uses natural language processing technology to analyze the standardized multimodal data and generate task requirement descriptions of user intentions and emotional states;

[0059] The task assignment module decomposes the task requirement description into a structured manner and assigns the decomposed subtasks to digital humans with corresponding capabilities;

[0060] Task execution module: each digital human retrieves historical context data and executes tasks according to the received subtasks, generating subtask execution results;

[0061] The result integration module adjusts the priority and output form of the task execution results according to the user's emotional state and generates integrated collaborative output results;

[0062] In the dynamic interaction module, based on the collaborative output results, the main digital human analyzes the user's real-time emotional fluctuations and dynamically adjusts the main digital human's output method.

[0063] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the virtual digital human interaction management method based on artificial intelligence as described in the first aspect of the present invention is implemented.

[0064] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the virtual digital human interaction management method based on artificial intelligence as described in the first aspect of the present invention.

[0065] The beneficial effects of the present invention are as follows: by introducing multimodal data fusion processing and emotional dynamic perception mechanism, the interaction effect between virtual digital humans and users is significantly improved; by standardized processing and feature extraction of user voice signals, facial expression data and gesture trajectory data, deep fusion of multimodal information and accurate analysis of user intentions are achieved; by adopting pre-trained natural language understanding models and long short-term memory networks (LSTM) and other technologies, the main intentions of users can be identified more accurately, and the execution priority of tasks can be dynamically adjusted through the emotion classification network, greatly improving the intelligent level of interaction; by real-time analysis of the user's emotional fluctuation data, the interaction tone, output form and content complexity of the main digital human can be dynamically adjusted, thereby achieving a more humane interaction method. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0067] Figure 1 This is a flow chart of the virtual digital human interaction management method based on artificial intelligence in Example 1.

[0068] Figure 2 Schematic diagram of the artificial intelligence virtual digital human interaction management system in Example 1. DETAILED DESCRIPTION

[0069] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0070] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0071] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0072] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides a virtual digital human interaction management method based on artificial intelligence, comprising the following steps:

[0073] S1. Collect multimodal data of users and perform standardization.

[0074] S1.1. Multimodal data includes the user's voice signal, facial expression data, and gesture trajectory data.

[0075] S1.2. Standardization processing includes noise reduction and feature extraction of the user's voice signal to generate a voice feature vector; normalization and feature enhancement of facial expression data to generate a facial expression feature vector; trajectory smoothing and time series sampling of gesture trajectory data to generate a gesture trajectory feature vector.

[0076] Specifically, the speech signal is processed using short-time Fourier transform (STFT), which is expressed as follows:

[0077] ;

[0078] in, Represents the speech signal at time and frequency The spectrum representation on is used to capture the time-frequency information of speech. Indicates the current time point, that is, the time position of the analysis, Indicates frequency, Represents the original speech signal over time Changing sound wave data, represents the time variable over the integral, Indicates at time The local time segment of the signal is captured above. Represents an imaginary unit.

[0079] Facial expression data processing is to normalize the facial key point detection and calculate the Euclidean distance of the key points to extract features. The expression is:

[0080] ;

[0081] in, represents the facial label feature vector, , Represents the two-dimensional coordinates of facial landmarks, i.e. the locations of specific facial features.

[0082] The expression for trajectory smoothing of gesture trajectory data is:

[0083] ;

[0084] in, Represents the smoothed gesture trajectory data, reduces noise interference, and retains the overall trend of the trajectory. represents the smoothing window size, represents the smoothing window time index, Represents the original gesture trajectory data at time The value of .

[0085] S2. Through natural language processing technology, the standardized multimodal data is analyzed to generate task requirement descriptions of user intentions and emotional states.

[0086] S2.1. Use the speech recognition model to perform speech-to-text processing on the speech feature vector to generate a text representation of the user's speech.

[0087] Specifically, text-to-text processing refers to mapping speech feature vectors into text sequences, that is, transcribing the user's speech content. Introducing a speech recognition model can perform context correction, which can effectively reduce the error rate of speech recognition and improve the accuracy of speech content.

[0088] S2.2. Perform word segmentation on the generated text, remove stop words and perform lemma restoration to obtain a word sequence.

[0089] Furthermore, word segmentation converts the text into an actionable word sequence, removes stop words, and reduces the interference of meaningless words on semantic analysis.

[0090] S2.3. Use the pre-trained natural language understanding model to semantically vectorize the word sequence and generate the semantic embedding vector of the text.

[0091] It should be noted that the use of pre-trained natural language understanding models can solve semantic ambiguity problems and improve the ability to understand complex sentences.

[0092] Semantic embedding vectors provide a deep representation of the semantics of user text, which can capture the semantic relationships between words, contextual meanings, and deep intentions.

[0093] S2.4. Use the classification model to classify the semantic embedding vector of the text and identify the user’s main intention.

[0094] It should be noted that accurately identifying the user's main intention provides a clear direction for task generation, reduces misunderstandings in task execution, and improves service quality.

[0095] S2.5. Use convolutional neural network to classify facial expression feature vectors and obtain the user's labeled emotion category; input the gesture trajectory feature vector as a time series into the long short-term memory network to extract the dynamic emotional features of the gesture and obtain the user's gesture emotion category; fuse the user's labeled emotion and gesture emotion to obtain the final emotional state; perform feature fusion of the semantic embedding vector and the final emotional state to generate a comprehensive feature vector; use the task generation model to infer the comprehensive feature vector and generate a structured task requirement description.

[0096] It should be noted that facial expression emotion classification directly captures the user's immediate emotions through visual signals, which supplements the deficiencies of voice and gesture emotions.

[0097] Gesture emotion analysis can capture the user's implicit action emotions during interaction, such as anger (waving hands quickly) and anxiety (shaking hands).

[0098] The emotion fusion strategy integrates the advantages of both facial and gesture emotion signals of users, avoiding the limitations of single-modal emotion analysis.

[0099] Through structured task descriptions, users' intentions and emotional states can be converted into machine-executable tasks.

[0100] S3. Decompose the task requirement description into a structured manner and assign the decomposed subtasks to digital humans with corresponding capabilities.

[0101] S3.1. Decompose the text part of the task requirement description through dependency parsing and extract the main verbs of the task.

[0102] Specifically, the main process of decomposition is to extract the main verbs in the task (such as "query", "play", "send", etc.) and the related dependent components (such as subject, object, modifiers, etc.) to form a semantic dependency tree; extract the core logical relationship of the task through the semantic dependency tree to generate the basic task unit (ie, task node).

[0103] It should be noted that dependency parsing can automatically extract core actions and related elements from complex task requirement descriptions, provide a clear semantic structure for subsequent task decomposition, improve the accuracy of task parsing, and reduce the need for human intervention.

[0104] S3.2. Structural modeling of tasks is performed based on the main verbs of the tasks to generate a task structure diagram.

[0105] Specifically, the task structure diagram refers to a task structure diagram in which the nodes represent specific subtasks (such as "obtain weather data"), and the edges represent the dependencies between tasks (such as "query the weather first, then notify the user"). The graph structure modeling method is used to define node attributes (such as task domain, priority) and edge attributes (such as task order, dependency strength).

[0106] S3.3. Embed the emotional state and context information in the task requirement description into the task structure diagram to obtain the final task structure diagram; establish a capability vector for each digital human in the collaborative network; for each task node in the final task structure diagram, calculate its matching degree with each digital human capability vector, expressed as:

[0107] ;

[0108] in, Represents a task node Human Ability Vector with Numbers The higher the matching degree, the more suitable the digital human is for executing the task node. Indicates A task node is a specific task unit in the task structure diagram. It is the task that needs to be assigned currently. It has domain attributes. Indicates the task node index, It represents the capability vector of a digital human, which describes the professional capabilities of a digital human in different task fields. The higher the capability value, the better the digital human is at this field. The capability vector of a digital human can be used to determine whether the digital human is suitable for the field requirements of executing the task node. Indicates that the task node is The importance of the task nodes in different fields reflects the demand intensity of the task nodes in different fields. fields (such as "weather query"), then the corresponding Higher, if a task is The dependence on one field is weak, then the corresponding Lower or 0, An index representing the task domain, represents the number of task areas, Indicates that digital people are in The ability value of each field is a component of the ability vector. The higher the ability value, the better the digital human is at handling tasks in this field. Represents the adjustment factor, which is used to balance the impact between weight and task complexity. Indicates the task area The complexity of the requirements, where a higher value means a more difficult task;

[0109] According to the matching degree, the task nodes are assigned to the digital human with the highest matching degree, and a task allocation table is generated; the tasks in the task allocation table are sent to the digital human with corresponding capabilities, and a collaborative mechanism is built to coordinate the order of data transmission between digital humans.

[0110] Specifically, the capability value of a digital human is obtained through evaluation of prior knowledge or training data. The higher the value, the better the digital human is at this field. For example:

[0111] Ability vector of digital human A: (Expert in NLP and data analysis).

[0112] Capability vector of digital human B: (Good at data analysis and visual processing).

[0113] Build a collaborative mechanism to coordinate the order of data transmission between digital humans. For example:

[0114] When the task After the weather query is completed, Digital Human A passes the result to Digital Human B for execution (Analyze data).

[0115] S4. Each digital human retrieves historical context data and executes tasks according to the received subtasks, generating subtask execution results.

[0116] S4.1. The digital human extracts contextual data related to the subtask from the task requirement description and retrieves it through the knowledge graph; based on the retrieved contextual data, it constructs API query parameters to obtain real-time data.

[0117] Specifically, the digital human retrieves information related to the subtask through the knowledge graph (KG). The specific steps are as follows:

[0118] Based on the context data, construct query conditions (such as SPARQL query statements); find relevant nodes and their attributes in the knowledge graph, for example:

[0119] When querying the weather, retrieve the nodes and edges related to "geographic location" and "weather type".

[0120] When querying recommended products, retrieve nodes related to "user preference" and "product category".

[0121] Extract task-related features (such as parameters of real-time data interfaces).

[0122] S4.2. Analyze the nodes related to the subtask in the knowledge graph and perform node reasoning through the graph neural network. The expression is:

[0123] ;

[0124] in, Representation Node in the The node feature representation of the layer. After reasoning at this layer, the semantic information and feature representation of the node are It represents the activation function that adds nonlinear characteristics to the result after linear transformation. Indicates The weight matrix of the layer is used to linearly transform the aggregated neighbor features. represents neighbor nodes, Representation Node The set of neighbor nodes of Representation and Node Connected neighbor nodes In the The feature representation of the layer, Represents the graph neural network in The bias vector of the layer is used to offset the feature transformation result. represents a node in the graph, Indicates the current number of layers in the graph neural network;

[0125] According to the task requirements, select the nodes related to the current reasoning task target, and filter out the user demand features through the query conditions in the knowledge graph, and use the node feature representation to score. The expression is:

[0126] ;

[0127] in, Representation Node The relevance score to the task user demand characteristics, Indicates the characteristics of user needs. represents the context weight, the influence of context information on the score, Represents context features, which are used to supplement user demand information.

[0128] S4.3. Sort the nodes based on the scores and generate recommended content based on the sorting results; the digital human adjusts the strategy based on the recommended content and screens out qualified solutions; based on the screened qualified solutions, generates personalized subtask execution results.

[0129] Specifically, based on the sorting results, generate recommended content related to the current task goal. For example:

[0130] For weather query tasks, relevant weather data nodes are recommended (such as "current temperature" and "future rainfall"); for product recommendation tasks, product nodes that meet user preferences are recommended (such as "users' frequently purchased products").

[0131] S5. Adjust the priority and output form of the task execution results according to the user's emotional state to generate an integrated collaborative output result.

[0132] Integrate the subtask execution results of each digital human into a preliminary structured data set; adjust the priority of each task result in the structured data set according to the user's current emotional state, sort them based on the priority, adjust the content and presentation form of the results, generate an integrated collaborative output result and pass it to the main digital human.

[0133] Specifically, the adjusted priority determines the sorting order of the task results. For example:

[0134] If the user's sentiment is "urgent", urgent tasks (such as "check the weather") will be given a higher priority.

[0135] If the user's emotion is "calm", the priority remains unchanged based on the task importance.

[0136] According to the adjusted task priorities, the result content and presentation format are optimized as follows:

[0137] Content Optimization:

[0138] Provide more detailed and accurate output for high priority tasks. For example:

[0139] For the “check weather” task, more details (such as air quality and wind speed) are added when the user is in an “eager” emotional state.

[0140] For low-priority tasks, only brief results are provided to reduce information redundancy.

[0141] Optimize presentation format:

[0142] High-priority tasks are displayed in a prominent manner (such as eye-catching colors and top display).

[0143] Optimize the display of results for different emotional states:

[0144] Urgent emotions: Use compact and intuitive formats, such as card-style displays.

[0145] Calm emotions: Use a complete format and show more details.

[0146] S6. Based on the collaborative output results, the master digital human analyzes the user's real-time emotional fluctuations and dynamically adjusts the master digital human's output method.

[0147] The master digital human receives the integrated collaborative output results from the collaborative network and compares them with the multimodal data collected in real time to detect the user's emotional fluctuations; based on the detection results, it updates the current emotional state and adjusts the master digital human's interactive tone, optimal output form, and the complexity of the output content; based on the adjusted tone, output form, and output content, the master digital human generates the final output content.

[0148] Specifically, the interactive tone of the main digital person is dynamically adjusted according to the updated emotional state:

[0149] Tone adjustment rules:

[0150] Urgent: The tone is quick, direct, and concise. Example: “Okay, here’s the weather information you requested.”

[0151] Calm: Gentle, detailed tone. Example: "Okay, here are the weather details you requested, including temperature, humidity, and air quality."

[0152] Anxiety: Use a soothing, reassuring tone. For example: "Don't worry, I've found a solution for you, check out the results below."

[0153] It should be noted that adjusting the tone enables the main digital person to be closer to the user's psychological state and enhance the affinity of the interaction. The appropriate tone soothes or motivates the user, helping to alleviate negative emotions or strengthen positive emotions.

[0154] Adjust the complexity and presentation of output based on emotional state:

[0155] Urgent emotions: Use concise and intuitive forms to reduce the amount of information, such as:

[0156] Display key results and highlight important data in card format (e.g., highlights).

[0157] Calm your emotions: Use detailed and complete forms to increase the amount of information, such as:

[0158] Display content in layers (e.g., collapsing sections for more details) and provide more context (e.g., additional instructions).

[0159] Anxiety: In a clear and easy to understand format such as:

[0160] Charts help explain complex data and present task results step by step, reducing information overload.

[0161] This embodiment also provides a virtual digital human interaction management system based on artificial intelligence, including:

[0162] The data collection module collects the user's multimodal data and performs standardization processing; the intention analysis module uses natural language processing technology to analyze the standardized multimodal data and generate a task requirement description of the user's intention and emotional state; the task assignment module performs structured decomposition of the task requirement description and assigns the decomposed subtasks to digital humans with corresponding capabilities; the task execution module, each digital human retrieves historical context data and executes tasks based on the received subtasks to generate subtask execution results; the result integration module adjusts the priority and output form of the task execution results according to the user's emotional state and generates an integrated collaborative output result; the dynamic interaction module, based on the collaborative output results, the main digital human analyzes the user's real-time emotional fluctuations and dynamically adjusts the main digital human's output method.

[0163] This embodiment also provides a computer device, which is suitable for the case of a virtual digital human interaction management method based on artificial intelligence, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the virtual digital human interaction management method based on artificial intelligence proposed in the above embodiment.

[0164] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0165] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for implementing the interaction management of a virtual digital human based on artificial intelligence proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0166] In summary, the present invention significantly improves the interaction effect between virtual digital humans and users by introducing multimodal data fusion processing and emotional dynamic perception mechanism; realizes deep fusion of multimodal information and accurate analysis of user intentions through standardized processing and feature extraction of user voice signals, facial expression data and gesture trajectory data; adopts pre-trained natural language understanding models and long short-term memory networks (LSTM) and other technologies to more accurately identify the main intentions of users, and dynamically adjusts the execution priority of tasks through the emotion classification network, greatly improving the intelligent level of interaction; and dynamically adjusts the interaction tone, output form and content complexity of the main digital human through real-time analysis of the user's emotional fluctuation data, thereby realizing a more humane interaction method.

[0167] Example 2, referring to the table below, is the second example of the present invention. To further verify the technical solution of the present invention, experimental simulation data of a virtual digital human interaction management method based on artificial intelligence are provided.

[0168] This experiment built a virtual digital human collaborative network system, including a main digital human and three sub-digital humans with different capabilities (digital human A, digital human B and digital human C). User interaction data is collected through voice input, facial expression capture and gesture tracking devices. Specifically, it includes:

[0169] Multimodal data processing:

[0170] The experiment first collects user voice, facial expression and gesture data, and generates voice feature vectors, facial expression feature vectors and gesture trajectory feature vectors respectively.

[0171] The speech feature vector is converted into text representation using a speech recognition model, and then a semantic embedding vector of the user's speech is generated through natural language processing techniques (such as word segmentation, lemma restoration, and semantic vectorization).

[0172] Facial expression data is used to classify emotions through a convolutional neural network, and gesture trajectory data is used to extract dynamic emotional features through a long short-term memory network, which are then fused to generate the final emotional state.

[0173] Task decomposition and allocation:

[0174] According to the user's voice content and emotional state, a structured task requirement description is generated, and the main verbs of the task are extracted through dependency parsing.

[0175] Construct a task structure diagram and embed the user's emotional state into the task nodes.

[0176] According to the matching degree calculation between the digital human capability vector and the task node, the subtasks after task decomposition are assigned to the most suitable digital human.

[0177] Task execution and result generation:

[0178] Based on the received subtask, the sub-digital human retrieves contextual data through the knowledge graph and uses graph neural network reasoning to generate personalized task execution results.

[0179] Each digital human sorts the task execution results according to priority and passes them to the main digital human.

[0180] Emotion adjustment and output optimization:

[0181] The master digital human receives the integrated task execution results and compares them with the user emotion data collected in real time to detect user emotion fluctuations.

[0182] Adjust the interaction tone, output content complexity and output form according to emotional fluctuations to generate the final output content.

[0183] The details of multimodal data collection are shown in Table 1 below:

[0184] Table 1 Multimodal data collection table

[0185]

[0186] Based on the feature vectors formed by multimodal data collection, a task structure diagram is constructed to obtain the capability vector of each digital human, as shown in Table 2 below:

[0187] Table 2 Digital human capability vector table

[0188]

[0189] Calculate the matching degree between the digital human capability vector and the task node. The expression is:

[0190] ;

[0191] The specific process is as follows:

[0192] Set the number of task areas , the weight vector , task complexity , adjustment factor .

[0193] Task Node (Check tomorrow's weather)

[0194] For the matching degree calculation of number A:

[0195] :

[0196] ;

[0197] :

[0198] ;

[0199] :

[0200] ;

[0201] Total match:

[0202] ;

[0203] Calculation of matching degree for digital person B:

[0204] For field 1:

[0205] ;

[0206] For field 2:

[0207] ;

[0208] For field 3:

[0209] ;

[0210] Total match:

[0211] ;

[0212] Calculation of matching degree of digital human C:

[0213] For field 1:

[0214] ;

[0215] For field 2:

[0216] ;

[0217] For field 3:

[0218] ;

[0219] Total match:

[0220] ;

[0221] Task Node T2 (Set Meeting Reminder)

[0222] Calculated in the same way, the final matching result is:

[0223] For number A:

[0224] ;

[0225] For number B:

[0226] ;

[0227] For number C:

[0228] ;

[0229] The matching results of each node are shown in Table 3 below:

[0230] Table 3. Matching degree between digital human capability vector and task node

[0231] Task Node Digital Human Matching results T1 Digital Human A 0.563 T1 Digital Human B 0.591 T1 Digital Human C 0.471 T2 Digital Human A 0.68 T2 Digital Human B 0.88 T2 Digital Human C 0.60

[0232] From the analysis of the above table, we can see that the task description of task node T1 is to query tomorrow's weather. The highest matching degree of digital human A in task node T1 is 0.85, so task node T1 is assigned to digital human A; in task node T2, the highest matching degree of digital human B is 0.88, so task node T2 is assigned to digital human B.

[0233] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A virtual digital human interaction management method based on artificial intelligence, characterized in that: include, Collect multimodal data from users and perform standardized processing; Through natural language processing technology, the standardized multimodal data is analyzed to generate task requirement descriptions of user intentions and emotional states; Decompose the task requirement description into a structured manner and assign the decomposed subtasks to digital humans with corresponding capabilities; Each digital human retrieves historical context data and executes tasks according to the received subtasks, generating subtask execution results; Adjust the priority and output form of task execution results according to the user's emotional state to generate integrated collaborative output results; Based on the collaborative output results, the master digital human analyzes the user's real-time emotional fluctuations and dynamically adjusts the master digital human's output method; The structural decomposition of the task requirement description and the allocation of the decomposed subtasks to digital humans with corresponding capabilities include the following steps: Decompose the text part of the task requirement description through dependency parsing to extract the main verbs of the task; Structural modeling of tasks is performed based on the main verbs of the tasks to generate a task structure diagram; The emotional state and context information in the task requirement description are embedded into the task structure diagram to obtain the final task structure diagram; Establish capability vectors for each digital human in the collaborative network; For each task node in the final task structure diagram, calculate its matching degree with each digital human capability vector; According to the matching degree, the task nodes are assigned to the digital human with the highest matching degree, and a task allocation table is generated; Send the subtasks in the task assignment table to digital humans with corresponding capabilities, and build a collaborative mechanism to coordinate the order of data transmission between digital humans.

2. The method for managing virtual digital human interaction based on artificial intelligence according to claim 1, characterized in that: The multimodal data includes a user's voice signal, facial expression data, and gesture trajectory data; The standardization process comprises the following steps: Perform noise reduction and feature extraction on the user's voice signal to generate a voice feature vector; Normalize and enhance the facial expression data to generate facial expression feature vectors; The gesture trajectory data is smoothed and time series sampled to generate the gesture trajectory feature vector.

3. The method for managing virtual digital human interaction based on artificial intelligence according to claim 2, characterized in that: Through natural language processing technology, the standardized multimodal data is parsed to generate task requirement descriptions of user intentions and emotional states, including the following steps: Use the speech recognition model to perform speech-to-text processing on the speech feature vector to generate a text representation of the user's speech; Perform word segmentation on the generated text, remove stop words and perform lemma restoration to obtain a word sequence; Use the pre-trained natural language understanding model to semantically vectorize the word sequence and generate the semantic embedding vector of the text; Use the classification model to classify the semantic embedding vector of the text and identify the user's main intention; The facial expression feature vector is classified by convolutional neural network to obtain the user's label emotion category; The gesture trajectory feature vector is input into the long short-term memory network as a time series to extract the dynamic emotional features of the gesture and obtain the user's gesture emotion category; The user's label emotion and gesture emotion are integrated to obtain the final emotional state; The semantic embedding vector and the final emotional state are fused to generate a comprehensive feature vector; The task generation model is used to reason over the comprehensive feature vector to generate a structured task requirement description.

4. The method for managing virtual digital human interaction based on artificial intelligence according to claim 3, characterized in that: Each digital human retrieves historical context data and executes tasks according to the received subtasks, and generates subtask execution results, including the following steps: The digital human extracts contextual data related to the subtask from the task requirement description and retrieves it through the knowledge graph; Based on the retrieved context data, construct API query parameters to obtain real-time data; Analyze the nodes related to the subtask in the knowledge graph and perform node reasoning through the graph neural network. The expression is: ； in, Representation Node in the The node feature representation of the layer, It represents the activation function that adds nonlinear characteristics to the result after linear transformation. Indicates The weight matrix of the layer, represents neighbor nodes, Representation Node The set of neighbor nodes of Representation and Node Connected neighbor nodes In the The feature representation of the layer, Represents the graph neural network in The bias vector of the layer, represents a node in the graph, Indicates the current number of layers in the graph neural network; According to the task requirements, select the nodes related to the current reasoning task target, and filter out the user demand features through the query conditions in the knowledge graph, and use the node feature representation to score. The expression is: ； in, Representation Node The relevance score to the task user requirement characteristics, Indicates the characteristics of user needs. represents the context weight, Represents contextual features; Sort the nodes based on the relevance scores and generate recommendations based on the sorting results; Digital people adjust strategies based on recommended content and screen out qualified solutions; Generate personalized subtask execution results based on the screened qualified solutions.

5. The method for managing virtual digital human interaction based on artificial intelligence according to claim 4, characterized in that: Adjusting the priority and output form of task execution results according to the user's emotional state and generating integrated collaborative output results include the following steps: Integrate the sub-task execution results of each digital human into a preliminary structured data set; According to the user's current emotional state, the priority of each task result in the structured data set is adjusted, and sorted based on the priority, the content and presentation form of the results are adjusted, and the integrated collaborative output results are generated and delivered to the main digital human.

6. The method for managing virtual digital human interaction based on artificial intelligence according to claim 5, characterized in that: Based on the collaborative output results, the master digital human analyzes the user's real-time emotional fluctuations and dynamically adjusts the master digital human's output mode, including the following steps: The master digital human receives the integrated collaborative output results from the collaborative network and compares them with the multimodal data collected in real time to detect the user's emotional fluctuations; Update the current emotional state according to the detection results, and adjust the interactive tone, output form and complexity of the output content of the main digital human; Based on the adjusted tone, output form and output content, the master digital human generates the final output content.

7. A virtual digital human interaction management system based on artificial intelligence, based on the virtual digital human interaction management method based on artificial intelligence according to any one of claims 1 to 6, characterized in that: include, Data collection module, which collects multimodal data of users and performs standardized processing; The intention analysis module uses natural language processing technology to analyze the standardized multimodal data and generate task requirement descriptions of user intentions and emotional states; The task assignment module decomposes the task requirement description into a structured manner and assigns the decomposed subtasks to digital humans with corresponding capabilities; Task execution module: each digital human retrieves historical context data and executes tasks according to the received subtasks, generating subtask execution results; The result integration module adjusts the priority and output form of the task execution results according to the user's emotional state and generates integrated collaborative output results; In the dynamic interaction module, based on the collaborative output results, the main digital human analyzes the user's real-time emotional fluctuations and dynamically adjusts the main digital human's output method.

Images