Intelligent man-machine interaction system based on element universe

Abstract

The invention relates to the technical field of man-machine interaction, and discloses an intelligent man-machine interaction system based on meta-universe, which is used for solving the problem of lagging caused by the fact that each virtual module possibly initiates response at the same time during man-machine interaction, and comprises the following steps: constructing a meta-universe scene, collecting multi-modal behavior information of a user in the meta-universe scene, and displaying the multi-modal behavior information of the user in the meta-universe scene. Performing unified formatting processing on the multi-modal behavior information to obtain formatted multi-modal behavior information, and performing joint reasoning on the formatted multi-modal behavior information by adopting a natural language processing engine and a multi-modal fusion model to obtain structured command data; the method comprises the following steps: obtaining a virtual module in a meta universe scene needing to make a response action according to structured command data, recording the virtual module as a driven module, obtaining priority information of all the driven modules, evaluating to obtain a priority index, and driving the modules to make the response action according to the priority index, thereby effectively improving the interaction response efficiency and the user experience continuity.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and more specifically to an intelligent human-computer interaction system based on a metaverse. Background Technology

[0002] With the continuous integration of virtual reality, artificial intelligence, and real-time rendering technologies, immersive human-computer interaction systems based on the metaverse are gradually being applied in complex scenarios such as digital education, remote operation and maintenance, virtual exhibitions, and industrial simulation. These systems construct a collaboratively accessible three-dimensional interactive space for multiple users, combining multimodal behavioral data such as voice, movement, gestures, and gaze to achieve natural interaction between users and the digital environment.

[0003] To meet the demands of understanding complex commands and real-time interaction, existing systems generally employ a multimodal semantic fusion mechanism. This mechanism uses a semantic recognition model to transform natural input into structured commands, which are then broadcast in parallel to multiple virtual modules within the scene that meet the specified conditions. Upon receiving the structured commands, each virtual module independently drives its own actions according to its internally defined response rules, enabling responses such as digital human voice feedback, virtual device state switching, interface navigation, or spatial manipulation.

[0004] However, the above-mentioned technologies have at least the following technical problems:

[0005] The existing metaverse interaction system lacks a centralized scheduling mechanism for multi-target response resources. When a user command triggers multiple semantic matching objects, each virtual module may initiate response operations simultaneously, leading to system resource competition, graphics rendering stuttering, or feedback logic conflicts, which reduces the efficiency of interaction response and the consistency of user experience. Summary of the Invention

[0006] In order to overcome the above-mentioned defects of the prior art, the present invention provides an intelligent human-computer interaction system based on the metaverse to solve the problems existing in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] An intelligent human-computer interaction system based on a metaverse includes: a metaverse scene generation and user mapping module for constructing a metaverse scene; a multimodal behavior acquisition module for collecting multimodal behavior information of the user in the metaverse scene, including natural interaction signals such as voice input, body movements, gesture trajectories, facial expressions, and gaze positions, and uniformly formatting the multimodal behavior information to obtain formatted multimodal behavior information; an intent recognition and semantic parsing module for jointly reasoning with a natural language processing engine and a multimodal fusion model on the formatted multimodal behavior information to obtain structured command data; and a priority index acquisition module for obtaining the required response action based on the structured command data. Virtual modules within the metaverse scenario are denoted as driven modules. Priority information for all driven modules is acquired, including keyword information, timestamp information, and user interaction information. A priority index is obtained based on this priority information. A virtual response control module drives the driven modules within the metaverse scenario to perform response actions based on the priority index. Pre-defined response rules are provided for each virtual module, including response action types, responsive structured command fields, and corresponding response semantic feature vectors. A human-computer interaction experience evaluation module collects user evaluations of the effects of virtual response control after its completion. An interaction experience coefficient is constructed based on the evaluation and transmitted to the research and analysis end.

[0009] Preferably, the construction steps of the metaverse scene are as follows: a metaverse construction template is preset; a topology scene configuration file is imported from the resource library; and a three-dimensional spatial framework is initialized to obtain a three-dimensional topology structure; on the three-dimensional topology structure, the interaction environment parameters of the metaverse scene are configured, and the interaction boundary between objects in the scene and the user's virtual image is established according to the interaction environment parameters; several virtual modules that can perform responsive actions are loaded into the scene; the functional semantic tags and control logic descriptions of each virtual module registered in the resource library are extracted, and a semantic tag index table is generated according to the module number; based on the user's login information, behavioral preferences, and permission settings, the appearance rendering, interactive behavior binding, and identity tag injection of the digital human model are completed to obtain a virtual image; the three-dimensional topology structure, interaction environment parameters, virtual modules, semantic tag index table, and virtual image are logically integrated to generate the metaverse scene.

[0010] Preferably, the steps for acquiring the structured command data are as follows: extracting multimodal behavior features from formatted multimodal behavior information, including speech content, gesture signals, action sequences, and gaze trajectories, to obtain the semantic expression of the multimodal behavior features; identifying the correlation between speech content, gesture signals, action sequences, and gaze trajectory behavior features through the modality alignment network in the multimodal fusion model, and constructing a fusion representation vector, which contains an intent semantic feature vector; calling a semantic classification model and a target recognition model to perform operation category classification and target matching reasoning on the fusion representation vector, respectively, to identify the operation type, operation method, and target object ID; analyzing the quantifiers, time words, spatial location vectors, and historical state caches in the fusion representation vector to generate corresponding operation parameters and context constraints; and organizing and formatting the operation type, operation method, target object ID, operation parameters, and context constraints to generate structured command data.

[0011] Preferably, the step of obtaining the semantic expression of multimodal behavioral features is as follows: inputting the speech content into a natural language processing engine to complete word segmentation, part-of-speech tagging and dependency analysis; inputting the gesture signal into a pre-trained gesture recognition network to extract the gesture category pre-pointing coordinates; inputting the action sequence into a temporal action recognition model to extract the action type and execution amplitude; and inputting the gaze trajectory into a gaze region recognition model to extract the current gaze object and gaze time window.

[0012] Preferably, the steps for obtaining the priority index are as follows: obtaining user voice content information, extracting keyword information based on the user voice content information, and evaluating a semantic matching coefficient based on the keyword information; obtaining timestamp information of the user issuing structured commands, and evaluating a contextual urgency coefficient based on the timestamp information of the user issuing structured commands; obtaining user interaction information, including the number of clicks, the number of visits, and the dwell time, and evaluating a user interaction preference coefficient based on the user interaction information; normalizing the semantic matching coefficient, the contextual urgency coefficient, and the user interaction preference coefficient, and then evaluating the semantic matching coefficient based on the normalized semantic matching coefficient. The priority index is obtained by evaluating the coefficients, context urgency coefficient, and user interaction preference coefficient. The specific steps are as follows: PY = a1 × SM' + a2 × US' + a3 × IP'; where PY represents the priority index, SM' represents the normalized semantic matching coefficient, US' represents the normalized context urgency coefficient, IP' represents the normalized user interaction preference coefficient, and a1, a2, and a3 represent the weight coefficients of the normalized semantic matching coefficient, the normalized context urgency coefficient, and the normalized user interaction preference coefficient, respectively.

[0013] Preferably, the steps for obtaining the semantic matching coefficient are as follows: In the intent recognition and semantic parsing module, the natural language processing engine is called to process the user's voice content, extract keywords of action or object meaning, and form a semantic keyword set; the set of responsive operation tags defined by each driven module during registration is obtained, and the set of responsive operation tags is converted into a semantic tag set; the semantic matching coefficient is calculated based on the semantic keyword set and the semantic tag set.

[0014] Preferably, the steps for obtaining the situation urgency coefficient are as follows: record the trigger timestamp and the current timestamp of the structured command issued by the user; calculate the difference between the current timestamp and the trigger timestamp to obtain the waiting response time of the driven module since the command was triggered in the current interaction round; obtain the maximum allowed response delay time preset by the driven module; query the state buffer in the virtual response control module to determine whether the current driven module is in an incomplete response state. If there are actions in progress, interface switching, animation rendering, etc. that have not yet ended, set the response state residual flag Sr = 1; otherwise, set it to Sr = 0; calculate the situation urgency coefficient based on the waiting response time, the maximum allowed response delay time, and the response state residual flag.

[0015] Preferably, the steps for obtaining the user interaction preference coefficient are as follows: counting the total number of click operations performed by the user on the driven module within a set period; counting the total number of sessions entered by the user in the scene related to the driven module within the set period, recorded as the number of access sessions; obtaining the cumulative dwell time of the user in the area of ​​the driven module; and calculating the user interaction preference coefficient based on the total number of click operations, the number of access sessions, and the cumulative dwell time.

[0016] Preferably, the step of driving the driven modules in the metaverse scene to make response actions according to the priority index is as follows: sort the priority index of each driven module in descending order, and drive the driven modules in the metaverse scene to make response actions in order.

[0017] Preferably, the step of constructing the interaction experience coefficient based on the effect evaluation is as follows: obtaining the timestamp of the user issuing the structured command, simultaneously obtaining the timestamp of the driven module actually starting to execute the response action, calculating the time difference between the user issuing the structured command and the driven virtual module actually starting to execute the response action based on the timestamp, and recording it as the interaction delay time difference; extracting the intent semantic feature vector in the fusion representation vector and the action semantic feature vector of the driven module in the preset response rules, calculating the cosine distance between the intent semantic feature vector and the action semantic feature vector and converting it into a deviation, recorded as the behavior consistency deviation; and calculating the interaction experience coefficient based on the interaction delay time difference and the behavior consistency deviation.

[0018] The technical effects and advantages of this invention are as follows:

[0019] A metaverse scenario is constructed, and multimodal behavioral information of users within the metaverse scenario is collected. This multimodal behavioral information is then uniformly formatted to obtain formatted multimodal behavioral information. A natural language processing engine and a multimodal fusion model are used to jointly infer structured command data from this formatted multimodal behavioral information. Based on this structured command data, virtual modules within the metaverse scenario that require a response are identified and designated as driven modules. Priority information for all driven modules is obtained, and a priority index is evaluated. Modules are then driven to perform response actions based on this priority index, effectively improving interaction response efficiency and user experience consistency. Attached Figure Description

[0020] Figure 1 This is a structural diagram of an intelligent human-computer interaction system based on the metaverse, provided for an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. In addition, the forms of the various structures described in the following embodiments are merely illustrative. The intelligent human-computer interaction system based on the metaverse involved in the present invention is not limited to the structures described in the following embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] This invention provides an intelligent human-computer interaction system based on the metaverse, such as... Figure 1 As shown, the system includes:

[0023] The Metaverse Scene Generation and User Mapping Module is used to construct metaverse scenes for users to engage in immersive interaction. The metaverse scene contains several virtual modules that can perform responsive actions and maps the user's entity identity to an interactive virtual avatar.

[0024] In this embodiment, it should be specifically explained that the construction steps of the metaverse scene are as follows:

[0025] Load the spatial topology resources used to construct the metaverse scene. These resources include a 3D coordinate system, basic terrain units, scene node connections, and physical boundary settings. A preset metaverse construction template is used. The topology scene configuration file corresponding to the current interactive application is imported from the resource library, and the 3D spatial framework is initialized to obtain the 3D topology, providing a structural foundation for the subsequent binding of content elements.

[0026] On a 3D topological structure, the interactive environment parameters of the metaverse scene are configured. These parameters include lighting models, material maps, collision rules, scene physical response parameters, and interactive area marking information. Based on these parameters, the interactive boundaries between objects within the scene and the user's virtual avatar are established, laying the physical and visual foundation for the virtual module to implement responsive actions.

[0027] Load several virtual modules that can perform responsive actions into the scene. The virtual modules include object instances with semantic function tags and responsive action definitions, such as virtual characters, interactive panels, device models, etc.

[0028] Extract the functional semantic tags and control logic descriptions registered in the resource library for each virtual module, and generate a semantic tag index table according to the module number. This table is used by the subsequent intent recognition and semantic matching modules when executing commands to ensure that the semantic driving path has a clear mapping.

[0029] Based on user login information, behavioral preferences, and permission settings, the digital human model undergoes appearance rendering, interactive behavior binding, and identity tag injection to obtain a virtual avatar. This virtual avatar possesses interactive positioning, perspective linkage, and command issuance capabilities, serving as the user's unique representative within the scene.

[0030] By logically integrating the three-dimensional topology, interactive environment parameters, virtual modules, semantic tag index table, and virtual avatars, a metaverse scene with spatial logical structure, responsive driving capabilities, and identity interaction interfaces is generated.

[0031] The multimodal behavior acquisition module is used to collect multimodal behavior information of users in the metaverse scene. The behavior information includes natural interaction signals such as voice input, body movements, gesture trajectories, facial expressions and gaze positions. The multimodal behavior information is uniformly formatted to obtain formatted multimodal behavior information.

[0032] The multimodal behavior acquisition module, as the core entry point for user input in this invention, is responsible for real-time acquisition of multi-source behavioral signals from the metaverse scene, including voice input, body movements, gesture trajectories, facial expressions, and gaze positions during natural interaction. These signals are then uniformly formatted to generate a standardized multimodal input data structure. This effectively solves the problems of incomplete understanding of user intent and limited response dimensions in traditional single-input methods. The system can integrate multiple sensory channels to perceive user intent, emotions, and attention distribution, providing a high-quality, multi-dimensional foundation of raw information for subsequent semantic parsing, intent recognition, and priority evaluation. This significantly improves the naturalness, accuracy, and immersive experience of human-computer interaction.

[0033] The intent recognition and semantic parsing module is used to jointly reason with a natural language processing engine and a multimodal fusion model to obtain structured command data from formatted multimodal behavioral information.

[0034] A Natural Language Processing (NLP) engine is a language intelligence processing module used to parse and understand the natural language content in user speech or text input. This engine transforms user input into structured language representations through techniques such as speech recognition, lexical analysis, syntactic parsing, named entity recognition, and intent recognition, extracting key semantic units such as operation instructions, target entities, and additional conditions. The NLP engine maps semantic information from the language layer to a system-recognizable interactive command layer, and is a core component of language understanding during intent recognition.

[0035] A multimodal fusion model is a fusion reasoning model used to collaboratively process multi-source behavioral data (including voice, gestures, actions, facial expressions, and gaze) generated by users during interactions, in order to achieve comprehensive recognition of interaction intent. This model constructs feature representation spaces for each modality and utilizes techniques such as feature alignment mechanisms, attention mechanisms, or graph neural networks to achieve semantic association and contextual joint reasoning between modalities, thereby generating structured output with consistent semantic intent. Multimodal fusion models can enhance the system's understanding ability in non-verbal interactions and are a key technical component supporting the parsing of complex behaviors in natural human-computer interaction scenarios.

[0036] In this embodiment, it should be specifically explained that the steps for obtaining structured command data are as follows:

[0037] Multimodal behavioral features are extracted from formatted multimodal behavioral information. These features include speech content, gesture signals, action sequences, and gaze trajectories. The semantic representation of these multimodal behavioral features is obtained by inputting the speech content into a natural language processing engine to perform word segmentation, part-of-speech tagging, and dependency analysis; simultaneously, gesture signals are input into a pre-trained gesture recognition network to extract gesture category pre-pointing coordinates; action sequences are input into a temporal action recognition model to extract action type and execution amplitude; and gaze trajectories are input into a gaze region recognition model to extract the current gaze object and gaze time window.

[0038] By using the modal alignment network in the multimodal fusion model, the correlation between speech content, gesture signals, action sequences and gaze trajectory behavioral features is identified, and a fusion representation vector is constructed. The fusion representation vector contains an intent semantic feature vector, which reflects the user's comprehensive intent state at a certain interaction moment and is used to support the subsequent semantic reasoning process.

[0039] The semantic classification model and the target recognition model are invoked to perform operation category classification and target matching reasoning on the fused representation vector, respectively, to identify the operation type (such as selection, movement, control), operation method (such as click, drag, activation), and target object ID, which serve as one of the core fields of the structured command data;

[0040] Analyze the quantifiers, time terms, spatial location vectors, and historical state cache in the fusion representation vector to generate corresponding operation parameters (such as direction, duration, amplitude, etc.) and context restrictions (such as only valid for objects within the current field of view, prohibiting overwriting previous states, etc.);

[0041] The operation type, operation method, target object ID, operation parameters, and context constraints are organized and formatted to generate structured command data. This data is defined using a standard data structure, facilitating rapid invocation and execution by downstream virtual response control modules. This structured command data, as the final output of the current round's interaction intent, is stored in the instruction queue for system scheduling.

[0042] The priority index acquisition module is used to obtain the virtual modules in the metaverse scene that need to take a response action based on the structured command data. These are referred to as driven modules. The priority information of all driven modules is obtained, including keyword information, timestamp information, and user interaction information. The priority index is obtained by evaluating the priority information.

[0043] In this embodiment, it should be specifically explained that the steps for obtaining the priority index are as follows:

[0044] Obtain user voice content information, extract keyword information based on user voice content information, and evaluate the semantic matching coefficient based on keyword information;

[0045] Obtain the timestamp information of the structured command issued by the user, and evaluate the situation urgency coefficient based on the timestamp information of the structured command issued by the user.

[0046] Acquire user interaction information, including the number of clicks, the number of visits, and the duration of dwell time, and evaluate the user interaction preference coefficient based on the user interaction information;

[0047] The semantic matching coefficient, contextual urgency coefficient, and user interaction preference coefficient are normalized. A priority index is then calculated based on these normalized coefficients. The specific steps for obtaining the priority index are as follows:

[0048] PY=a1×SM'+a2×US'+a3×IP';

[0049] In the formula, PY represents the priority index, SM' represents the normalized semantic matching coefficient, which reflects the similarity between the user's current operation target and the registered semantic tags of the virtual module. When the matching degree is high, it means that the module best matches the user's intention, and the system should prioritize driving the response to it, thereby ensuring the semantic accuracy and relevance of the interaction results. This proportional relationship helps to improve the intelligent understanding and execution efficiency of human-computer interaction and avoid the response deviating from the user's original intention. US' represents the normalized situational urgency coefficient. When the user's operation has been triggered but is still in a waiting state, or when the module has unfinished residual behavior, the system quantifies the urgency of its response through the situational urgency coefficient, thereby prioritizing the scheduling and execution of modules with high time sensitivity. This direct proportional relationship ensures that the system prioritizes operations that have the greatest impact on the continuity of interaction and user perception in multi-task concurrent scenarios, effectively reducing response latency and the risk of interaction interruption, and improving the overall smoothness of interaction. IP' represents the normalized user interaction preference coefficient. When a module has a high click rate, access frequency or behavior persistence characteristics in the user's past use, the system generates a high preference coefficient accordingly, indicating that the module has a high degree of attention or usage priority for the user. This proportional relationship endows the response mechanism with personalized characteristics, helping to improve the alignment between interactive content and user expectations, thereby enhancing the system's human-computer adaptability and the quality of interactive experience. a1, a2, and a3 represent the weight coefficients of the normalized semantic matching coefficient, the normalized situational urgency coefficient, and the normalized user interaction preference coefficient, respectively, with a1 + a2 + a3 = 1. a1, a2, and a3 are obtained through the Analytic Hierarchy Process (AHP). For example, a1, a2, and a3 can be 0.3, 0.4, and 0.3, respectively. AHP is a system analysis method used to determine weights in multi-factor decision-making problems. Its basic principle is to decompose complex decision problems into target, criterion, and alternative layers. By constructing pairwise comparison matrices, the relative importance of each factor is judged pairwise, and then the relative weights of each factor are calculated. This method ensures the rationality of the judgment results through consistency checks, effectively quantifying expert experience or internal system rules, and providing a structured and controllable basis for weight allocation in the comprehensive calculation of multiple indicators. In this embodiment, the analytic hierarchy process (AHP) is used to determine the weight ratio of semantic matching coefficient, contextual urgency coefficient, and user interaction preference coefficient when calculating the priority index, ensuring that the priority evaluation results are objective and structurally reasonable.

[0050] In this embodiment, it should be specifically explained that the steps for obtaining the semantic matching coefficient are as follows:

[0051] In the intent recognition and semantic parsing module, the natural language processing engine is called to process the user's voice content, extract keywords with action or object meaning, and form a set of semantic keywords. Keywords include verbs (such as "open" and "zoom in"), nouns (such as "image" and "video"), or high-frequency control words set by the system.

[0052] Obtain the set of responsive operation tags defined by each driven module during registration, and transform the set of responsive operation tags into a set of semantic tags. The set of semantic tags typically includes predefined actions (such as "play" and "adjust") and object entities (such as "light" and "panel").

[0053] The semantic matching coefficient is calculated based on the set of semantic keywords and the set of semantic tags. The specific steps are as follows:

[0054]

[0055] In the formula, SM represents the semantic matching coefficient, and K c Represented as a set of semantic keywords, K m Represented as a set of semantic tags, the matching degree is evaluated from two perspectives: "command covers module tags" and "module tags cover commands," based on the intersection coverage between the set of semantic keywords extracted from structured commands and the set of preset semantic tags in virtual modules. By constructing a product relationship between the two coverage rates, the problem of bias amplification in one direction can be effectively avoided, ensuring that the matching results reflect both the degree to which the user's intent matches the module's function and the module's ability to adapt to the user's expression.

[0056] In this embodiment, it should be specifically explained that the steps for obtaining the situation urgency coefficient are as follows:

[0057] Record the trigger timestamp of the structured command issued by the user, and the timestamp serves as the starting point for determining the response waiting time;

[0058] Get the current timestamp, which indicates the point in time when response priority determination will be performed;

[0059] Calculate the difference between the current timestamp and the trigger timestamp to obtain the waiting response time of the driven module since the command was triggered in the current interaction round; obtain the maximum allowed response delay time preset by the driven module;

[0060] Query the status buffer in the virtual response control module to determine whether the currently driven module is in an incomplete response state. If there are actions, interface switching, animation rendering, or other behaviors that have not yet ended, set the response state residual flag Sr=1; otherwise, set it to Sr=0.

[0061] The situation urgency coefficient is calculated based on the waiting time for response, the maximum allowable response delay time, and the residual flag of the response status. The specific steps for obtaining the coefficient are as follows:

[0062]

[0063] In the formula, US represents the situation urgency coefficient, and T w T represents the waiting time for a response. exp The maximum allowable response delay time is represented by Sr, which represents the residual response status flag. A waiting time term is constructed based on the difference between the user command trigger time and the current time. This is combined with the virtual module's preset response timeout threshold and whether the module currently has a residual response status. The waiting ratio is compressed using a logarithmic function and then superimposed with the residual status flag to quantitatively characterize the urgency of the response. The formula's principle reflects the evaluation logic of "the longer the wait, the more urgent; the unclear the status, the more urgent," avoiding abstract priority settings or static rule judgments.

[0064] In this embodiment, it should be specifically explained that the steps for obtaining the user interaction preference coefficient are as follows:

[0065] The system counts the total number of clicks performed by users on the driven module within a set period. Clicks include behaviors that the system recognizes as having a clear interactive intent, such as voice activation, action triggering, and button clicks.

[0066] The number of times a user enters a session related to the scenario where the driven module is located within a set period is counted and recorded as the number of access sessions.

[0067] The cumulative dwell time of users within the driven module area is obtained. Dwell time refers to the total duration for which the user's virtual avatar or gaze focuses within the module's space. It is a behavioral indicator that measures the degree of sustained user attention.

[0068] The user interaction preference coefficient is calculated based on the total number of clicks, the number of access sessions, and the cumulative dwell time. The specific steps to obtain the coefficient are as follows:

[0069]

[0070] In the formula, IP represents the user interaction preference coefficient, and N c N represents the total number of clicks. s T represents the number of access sessions. d This is expressed as cumulative dwell time. By comprehensively considering users' historical click counts, access frequency, and dwell time for virtual modules, a preference score index is constructed. Square root and normalization methods are used to quantify users' attention to and usage preferences for each module. The formula reflects the preference logic of "more frequent clicks, longer dwell time, and more concentrated single uses."

[0071] The virtual response control module drives the driven modules in the metaverse scene to make response actions according to the priority index. The response actions include voice feedback, action feedback, object operation or interface switching operation types. It also presets response rules for each virtual module. The response rules include response action type, respondable structured command fields and corresponding response semantic feature vectors. The response semantic feature vectors are used to represent the expression form of the module's response action in a space.

[0072] In this embodiment, it should be specifically explained that the steps for driving the driven modules in the metaverse scene to perform response actions based on the priority index are as follows:

[0073] The priority index of each driven module is sorted from high to low, and the driven modules in the metaverse scene are driven to make response actions in order.

[0074] By prioritizing driven modules within the metaverse scenario according to a priority index and driving responses sequentially, the system effectively resolves resource conflicts and response latency issues under multi-objective concurrent responses. By introducing a priority index, the system can dynamically evaluate the response value of each module to the current user intent based on objective factors such as semantic matching degree, contextual urgency, and user preferences. This prioritizes activating the driven modules that have the greatest impact on the interaction outcome, achieving orderly scheduling of the response process and optimal resource utilization. This mechanism not only improves the accuracy and timeliness of the system's response but also enhances the user's operational coherence and immersive experience in complex interaction scenarios, resulting in a significant improvement in interaction quality.

[0075] The human-computer interaction experience evaluation module is used to collect the effect evaluation of the user's implementation of virtual response control after the virtual response control is completed. The effect evaluation includes interaction latency and behavior consistency indicators. Based on the effect evaluation, the interaction experience coefficient is constructed and transmitted to the research and analysis end for subsequent improvement of semantic recognition model, adjustment of module priority index weight or optimization of virtual module behavior script, thereby achieving continuous improvement of system interaction performance.

[0076] In this embodiment, it should be specifically explained that the steps for constructing the interaction experience coefficient based on the effect evaluation are as follows:

[0077] Get the timestamp of the structured command issued by the user, and simultaneously get the timestamp of the actual start of the response action of the driven module. Calculate the time difference between the time the user issues the structured command and the actual start of the response action of the driven virtual module based on the timestamps, and record it as the interaction delay time difference.

[0078] Extract the intent semantic feature vector from the fused representation vector and the action semantic feature vector of the driven module in the preset response rules. Calculate the cosine distance between the intent semantic feature vector and the action semantic feature vector and convert it into a deviation, denoted as the behavior consistency deviation. The specific steps are as follows:

[0079]

[0080] In the formula, DB represents the behavioral consistency deviation, and V u Represented as an intent semantic feature vector, V r It is represented as an action semantic feature vector, and the larger the behavior consistency deviation value, the larger the behavior deviation;

[0081] The interaction experience coefficient is calculated based on the interaction latency time difference and behavior consistency deviation. The specific steps to obtain the coefficient are as follows:

[0082] EH = ln(TD+1)·(DB+1);

[0083] In the formula, EH represents the interaction experience coefficient, TD represents the interaction latency time difference, and DB represents the behavior consistency deviation. Based on the product relationship between user operation latency and behavior consistency deviation, the natural logarithm function is used to compress and transform the joint effect, reflecting the combined impact of the two key factors, "timeliness of response" and "accuracy of response," on the overall interaction experience. By quantifying the interaction latency time and semantic deviation separately and multiplying them, a single dimension can be avoided from dominating the score. At the same time, it is ensured that when both aspects are poor, the interaction experience coefficient increases significantly, effectively reflecting the decline in system performance. Furthermore, the compression processing using the natural logarithm function can suppress the amplification effect of extreme values ​​on the scoring results, making the scoring results more stable and highly distinguishable.

[0084] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0085] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An intelligent human-computer interaction system based on a metaverse, characterized in that, The system includes; The Metaverse Scene Generation and User Mapping module is used to construct metaverse scenes; The multimodal behavior acquisition module is used to collect multimodal behavior information of users in the metaverse scene. The behavior information includes natural interaction signals such as voice input, body movements, gesture trajectories, facial expressions and gaze positions. The multimodal behavior information is uniformly formatted to obtain formatted multimodal behavior information. The intent recognition and semantic parsing module is used to jointly reason with a natural language processing engine and a multimodal fusion model to obtain structured command data from formatted multimodal behavioral information. The priority index acquisition module is used to obtain the virtual modules in the metaverse scene that need to take a response action based on the structured command data. These are referred to as driven modules. The priority information of all driven modules is obtained, including keyword information, timestamp information, and user interaction information. The priority index is obtained by evaluating the priority information. The virtual response control module drives the driven modules in the metaverse scene to make response actions according to the priority index, and presets response rules for each virtual module. The response rules include response action type, respondable structured command fields and corresponding response semantic feature vectors. The human-computer interaction experience evaluation module is used to collect the effect evaluation of the user after the virtual response control is completed, construct the interaction experience coefficient based on the effect evaluation, and transmit the interaction experience coefficient to the research and analysis terminal.

2. The intelligent human-computer interaction system based on the metaverse according to claim 1, characterized in that: The steps for constructing the metaverse scene are as follows: The pre-defined metaverse construction template is used to import the topology scene configuration file from the resource library and initialize the three-dimensional spatial framework to obtain the three-dimensional topology structure. On a three-dimensional topological structure, configure the interaction environment parameters of the metaverse scene, and establish the interaction boundary between objects in the scene and the user's virtual image based on the interaction environment parameters; Load several virtual modules that can perform responsive actions into the scene; Extract the functional semantic tags and control logic descriptions registered in the resource library for each virtual module, and generate a semantic tag index table according to the module number; Based on user login information, behavioral preferences, and permission settings, the appearance rendering, interactive behavior binding, and identity tag injection of the digital human model are completed to obtain a virtual image. By logically integrating the 3D topology, interactive environment parameters, virtual modules, semantic tag index table, and virtual avatars, a metaverse scene is generated.

3. The intelligent human-computer interaction system based on the metaverse according to claim 1, characterized in that, The steps for obtaining the structured command data are as follows: Multimodal behavioral features are extracted from formatted multimodal behavioral information. These features include speech content, gesture signals, action sequences, and gaze trajectories. The semantic representation of the multimodal behavioral features is then obtained. By using the modal alignment network in the multimodal fusion model, the correlation between speech content, gesture signals, action sequences, and gaze trajectory behavioral features is identified, and a fusion representation vector is constructed, which contains an intent semantic feature vector. The semantic classification model and the target recognition model are invoked to perform operation category classification and target matching inference on the fused representation vector, respectively, to identify the operation type, operation method and target object ID; Analyze the quantifiers, time terms, spatial location vectors, and historical state cache in the fusion representation vector to generate corresponding operation parameters and context constraints; The operation type, operation method, target object ID, operation parameters, and context constraints are organized and formatted to generate structured command data.

4. The intelligent human-computer interaction system based on the metaverse according to claim 3, characterized in that, The steps for obtaining the semantic representation of multimodal behavioral features are as follows: The speech content is input into the natural language processing engine to complete word segmentation, part-of-speech tagging, and dependency analysis. The gesture signal is input into a pre-trained gesture recognition network to extract the gesture category pre-pointing coordinates; Input the action sequence into the temporal action recognition model to extract the action type and execution magnitude; Input the gaze trajectory into the gaze region recognition model to extract the current gaze object and gaze time window.

5. The intelligent human-computer interaction system based on the metaverse according to claim 1, characterized in that: The steps for obtaining the priority index are as follows: Obtain user voice content information, extract keyword information based on user voice content information, and evaluate the semantic matching coefficient based on keyword information; Obtain the timestamp information of the structured command issued by the user, and evaluate the situation urgency coefficient based on the timestamp information of the structured command issued by the user. Acquire user interaction information, including the number of clicks, the number of visits, and the duration of dwell time, and evaluate the user interaction preference coefficient based on the user interaction information; The semantic matching coefficient, contextual urgency coefficient, and user interaction preference coefficient are normalized. A priority index is then calculated based on these normalized coefficients. The specific steps for obtaining the priority index are as follows: PY=a1×SM'+a2×US'+a3×IP'; In the formula, PY represents the priority index, SM' represents the normalized semantic matching coefficient, US' represents the normalized context urgency coefficient, IP' represents the normalized user interaction preference coefficient, and a1, a2, and a3 represent the weight coefficients of the normalized semantic matching coefficient, the normalized context urgency coefficient, and the normalized user interaction preference coefficient, respectively.

6. The intelligent human-computer interaction system based on the metaverse according to claim 5, characterized in that: The steps for obtaining the semantic matching coefficients are as follows: In the intent recognition and semantic parsing module, the natural language processing engine is called to process the user's voice content, extract keywords of action or object meaning, and form a set of semantic keywords; Obtain the set of responsive operation tags defined by each driven module during registration, and convert the set of responsive operation tags into a set of semantic tags; The semantic matching coefficient is calculated based on the set of semantic keywords and the set of semantic tags.

7. The intelligent human-computer interaction system based on the metaverse according to claim 5, characterized in that: The steps for obtaining the situation urgency coefficient are as follows: Record the trigger timestamp and current timestamp of the structured command issued by the user; The difference between the current timestamp and the trigger timestamp is calculated to obtain the waiting response time of the driven module since the command was triggered in the current interaction round; Obtain the preset maximum allowable response delay time of the driven module; Query the status buffer in the virtual response control module to determine whether the currently driven module is in an incomplete response state. If there are actions, interface switching, animation rendering, or other behaviors that have not yet ended, set the response state residual flag Sr=1; otherwise, set it to Sr=0. The urgency coefficient of the situation is calculated based on the waiting time for response, the maximum allowable response delay time, and the residual flag of the response status.

8. The intelligent human-computer interaction system based on the metaverse according to claim 5, characterized in that: The steps for obtaining the user interaction preference coefficient are as follows: Count the total number of clicks performed by users on the driven module within a set period; The number of times a user enters a session related to the scenario where the driven module is located within a set period is counted and recorded as the number of access sessions. Obtain the user's cumulative dwell time within the driven module area; The user interaction preference coefficient is calculated based on the total number of clicks, the number of access sessions, and the cumulative dwell time.

9. The intelligent human-computer interaction system based on the metaverse according to claim 1, characterized in that: The steps for driving the driven modules within the metaverse scene to perform response actions based on the priority index are as follows: The priority index of each driven module is sorted from high to low, and the driven modules in the metaverse scene are driven to make response actions in order.

10. The intelligent human-computer interaction system based on the metaverse according to claim 1, characterized in that: The steps for constructing the interaction experience coefficient based on the effect evaluation are as follows: Get the timestamp of the structured command issued by the user, and simultaneously get the timestamp of the actual start of the response action of the driven module. Calculate the time difference between the time the user issues the structured command and the actual start of the response action of the driven virtual module based on the timestamps, and record it as the interaction delay time difference. Extract the intent semantic feature vector from the fusion representation vector and the action semantic feature vector of the driven module in the preset response rules. Calculate the cosine distance between the intent semantic feature vector and the action semantic feature vector and convert it into a deviation, which is denoted as behavior consistency deviation. The interaction experience coefficient is calculated based on the interaction delay time difference and the behavioral consistency deviation.

Images