An intelligent interaction method and system based on metaverse

Abstract

The present invention provides a metaverse-based intelligent interaction method and system thereof, and relates to the field of intelligent interaction technology. The method comprises: receiving a multimodal input signal of a user, and first preprocessing the multimodal input signal to generate multimodal feature data; obtaining background information of the user, and preprocessing the background information to generate background feature data; fusing the preprocessed multimodal feature data with the background feature data to generate input data; calculating the user's true intention according to the input data; dynamically adjusting the interactive content and presentation method in a pre-established metaverse virtual environment according to the user's true intention; and optimizing the interactive content in real time according to the user's feedback information. The present invention improves the personalization of interaction with the user and can accurately identify the user's true intention.

Description

Technical Field

[0001] The present invention relates to the field of intelligent interaction technology, and in particular to an intelligent interaction method and system based on a metaverse. Background Art

[0002] The current Metaverse interactive system mainly relies on the traditional graphical user interface, combined with simple gesture recognition, voice input and other multimodal interaction methods to realize the interaction between users and virtual environments. These systems usually use rule-driven or shallow semantic analysis technology to make a preliminary understanding of the user's input voice, text and actions, and generate corresponding interactive responses. At the same time, some systems have certain context-aware capabilities and can adjust the interaction method in specific situations, but the application of these technologies is still relatively limited.

[0003] Existing metaverse interaction technologies have significant deficiencies in understanding user expressions. First, the semantic fusion capability of multimodal input is weak, making it difficult to accurately identify and interpret complex user expressions. Second, existing technologies have limited ability to understand user intent and context, especially in dynamic situations. The system cannot accurately infer the user's true intent, resulting in misunderstandings or inaccurate responses during the interaction process.

[0004] Therefore, how to provide an intelligent interaction method and system based on the metaverse is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a subject matter that aims to solve the problems mentioned in the background technology.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0007] In a first aspect, a metaverse-based intelligent interaction method is provided, the method comprising the following steps:

[0008] Receiving a multimodal input signal from a user, and first preprocessing the multimodal input signal to generate multimodal feature data, wherein the multimodal input signal includes voice, gesture, and facial expression;

[0009] Obtaining user background information, and preprocessing the background information to generate background feature data;

[0010] The preprocessed multimodal feature data is fused with the background feature data to generate input data;

[0011] Through the convolutional neural network, the input data is used to extract speech features, gesture features and facial expression features;

[0012] Arrange the speech features, gesture features and facial expression features respectively according to the time series order to obtain a speech feature matrix, a gesture feature matrix and a facial expression feature matrix;

[0013] According to the speech feature matrix, gesture feature matrix and facial expression feature matrix, a multi-layer semantic network structure is constructed;

[0014] Through the cross-modal alignment formula, semantic mapping between different modal features is achieved to obtain a semantic feature matrix;

[0015] According to the semantic feature matrix and background feature data, the user's true intention is calculated;

[0016] Dynamically adjust the interactive content and presentation method in the pre-established metaverse virtual environment based on the user's real intention;

[0017] Optimize interactive content in real time based on user feedback.

[0018] Further, performing a first preprocessing on the multimodal input signal to generate multimodal feature data includes:

[0019] Performing a second preprocessing on the multimodal input signal to obtain initial multimodal feature data;

[0020] Perform time series analysis on the initial multimodal feature data to determine the synchronization and priority between the feature data of each modality;

[0021] Determine the weight coefficients between the feature data of each modality according to the synchronization and priority between the feature data of each modality;

[0022] According to the weight coefficients between the feature data of each modality, the feature data of each modality are fused through a fusion formula to obtain multimodal feature data.

[0023] Furthermore, the initial multimodal feature data is subjected to time series analysis to determine the synchronization and priority between the modal signals, including:

[0024] Decomposing the initial multimodal feature data into a state sequence for representing changes of each modal feature data at different time points;

[0025] According to all state sequences, the joint probability distribution of each modal feature data is calculated, and the joint probability distribution is used to determine the transition probability between each state sequence and the conditional probability of each modal feature data appearing under each state sequence; the calculation formula of the joint probability distribution is as follows:

[0026] ,

[0027] in, is the joint probability distribution of each modal feature data sequence corresponding to all state sequences, X is the set of each modal feature data sequence corresponding to all state sequences, is the state sequence at time t, is the characteristic data of each mode at time t, At time t=1, the initial state sequence is entered The probability of is the state sequence From the previous state sequence The transition probability obtained by the transfer, For the state sequence Below, each modal characteristic data The conditional probability of occurrence, arrive For each state in the state sequence, is the total length of the time series;

[0028] According to the joint probability distribution, the synchronization between the state sequences of the initial multimodal feature data is analyzed to determine the feature data combination;

[0029] According to the combination of feature data, combined with the transition probability between state sequences and the conditional probability of the initial multimodal feature data under each state sequence, the priority of each modal feature data is determined.

[0030] Furthermore, according to the feature data combination, combined with the transition probability between state sequences and the conditional probability of the initial multimodal feature data under each state sequence, the priority of each modal feature data is determined, including:

[0031] According to the characteristic data combination, combined with the transition probability of each state sequence and the conditional probability of the modal characteristic data under each state sequence, the stability coefficient of each state sequence is calculated. The calculation formula is as follows:

[0032] ,in, is the stability coefficient of the state sequence;

[0033] According to the stability coefficient of each state sequence And the conditional probability of each modal feature data under the state sequence , calculate the influence of each modal characteristic data , influence The calculation formula is as follows:

[0034] ,in, represents the influence of the i-th modal feature data, s represents all possible state sequences, To sum all possible state sequences s and integrate the influence of each state on the modal characteristic data, is the i-th modal characteristic data at time t;

[0035] According to the influence of each modal characteristic data obtained by calculation , determine the priority of each modal feature data , priority The calculation formula is as follows:

[0036] ,in, is the priority of the i-th modal feature data, is the influence of the i-th modal feature data, is the sum of the influences of all modal feature data, m is the number of modal feature data, is the influence of the j-th modal feature data.

[0037] Furthermore, the cross-modal alignment formula is as follows:

[0038] ,

[0039] in, is the aligned semantic feature matrix, is the speech feature matrix, is the gesture feature matrix The transpose of is the facial expression feature matrix, d is the dimension of the feature vector, is the normalization function;

[0040] The output of the user's true intention is achieved through the following calculation formula:

[0041] ,

[0042] in, The output of the user's true intention. is the background feature data, is the semantic feature matrix, is the weighted coefficient between the background feature data and the semantic feature matrix, is the normalization function.

[0043] Furthermore, according to the user's real intention, the interactive content and presentation method are dynamically adjusted in the pre-established metaverse virtual environment, including:

[0044] Generate preliminary interactive content based on the user's real intention;

[0045] The preliminary interactive content is scored according to the scoring formula to obtain an interactive content scoring set; the calculation formula for the interactive content scoring is:

[0046] ,

[0047] in, For the initial interactive content rating, is the influence of the i-th characteristic factor on user u and content c, is the weight of the i-th feature factor, and n is the number of feature factors.

[0048] Furthermore, according to the user's real intention, the interactive content and presentation method are dynamically adjusted in the pre-established metaverse virtual environment, including:

[0049] Obtain user behavior history scores, content relevance scores, and emotional state scores;

[0050] The interactive content score set is sorted by multi-dimensional priority according to the user's feedback information to obtain an interactive content priority score set; the calculation formula for the interactive content priority is as follows:

[0051] ,

[0052] in, Score the priority of content c for user u, Score user behavior history, Score the user's emotional state, Score the relevance of the content. Rate the situation. is the feedback correction coefficient, They are the weight coefficients of the behavior history score and the affective state score, the content relevance score, the situation score, and the feedback correction coefficient;

[0053] Generate interactive content based on the results of multi-dimensional priority sorting.

[0054] Furthermore, based on user feedback, the interactive content is optimized in real time, including:

[0055] Obtaining real-time feedback information from users and performing a third preprocessing on the information to obtain real-time feedback data;

[0056] Fusion of real-time feedback data and historical feedback data to generate a multi-dimensional feedback feature matrix;

[0057] The multi-dimensional feedback feature matrix is ​​subjected to a fourth preprocessing, and the parameters for generating the interactive content are adjusted to obtain the adjustment parameters; the calculation formula of the adjustment parameters is:

[0058] ,

[0059] in, To adjust the parameters for generating interactive content, Parameters for generating interactive content, is the learning rate, which determines the step size of each parameter update. is the loss function based on feedback data About parameters The gradient of To balance the coefficient, adjust the influence weight of different loss functions in the final parameter update. is the loss function based on the metaverse virtual environment structure About parameters gradient.

[0060] Furthermore, based on user feedback, the interactive content is optimized in real time, including:

[0061] The feedback data is divided into three categories: immediate feedback, periodic feedback and long-term feedback;

[0062] According to the classification results of the feedback data, the interactive content is adjusted in multiple dimensions. The calculation formula for the multi-dimensional weight adjustment is:

[0063] ,in, is the adjusted weight, is the impact weight of the i-th type of feedback data, is the weight coefficient of each type of feedback data.

[0064] In a second aspect, an intelligent interaction system based on the metaverse is provided, the system comprising:

[0065] A multimodal information processing module, used to receive a multimodal input signal from a user, and first pre-process the multimodal input signal to generate multimodal feature data, wherein the multimodal input signal includes voice, gesture and facial expression;

[0066] A background information processing module is used to obtain the user's background information, pre-process the background information, and generate background feature data;

[0067] A data fusion module is used to fuse the pre-processed multi-modal feature data with the background feature data to generate input data;

[0068] The feature extraction module is used to extract speech features, gesture features and facial expression features from the input data through a convolutional neural network;

[0069] A feature matrix building module is used to arrange the speech features, gesture features and facial expression features according to the time series order to obtain a speech feature matrix, a gesture feature matrix and a facial expression feature matrix;

[0070] A multi-layer semantic network structure building module is used to build a multi-layer semantic network structure according to a speech feature matrix, a gesture feature matrix and a facial expression feature matrix;

[0071] A semantic feature matrix generation module is used to achieve semantic mapping between different modal features through a cross-modal alignment formula to obtain a semantic feature matrix;

[0072] The real intention generation module is used to calculate the user's real intention based on the semantic feature matrix and background feature data;

[0073] A dynamic adjustment module is used to dynamically adjust the interactive content and presentation method in the pre-established metaverse virtual environment according to the user's real intention;

[0074] The feedback optimization module is used to optimize the interactive content in real time according to the user's feedback information.

[0075] The above solution of the present invention includes at least the following beneficial effects:

[0076] By receiving multimodal input signals and preprocessing them to generate multimodal feature data, the interaction effect between users and virtual environments can be significantly improved. The multimodal input signals mentioned include voice, gestures, and facial expressions. These signals are processed by the multimodal fusion algorithm to extract more dimensional information from a single user input, thereby improving the accurate understanding of user intentions. Combined with the extracted relevant background information, the system's situational awareness of users can be further enhanced, especially in complex virtual scenes. It can respond to user operations in real time and dynamically adjust the interactive content and presentation methods in the virtual environment, thereby providing a personalized interactive experience.

[0077] This intelligent interaction method based on semantic understanding of virtual environment can accurately identify the user's true intentions and make timely adjustments in the virtual environment, avoiding the common interaction inaccuracy caused by a single input method in existing technologies. In addition, by real-time monitoring of user feedback information and making adaptive adjustments, the system can continuously optimize its own response capabilities to ensure that user intentions are accurately responded to, thereby significantly improving the user's immersion and satisfaction in the metaverse. Compared with existing technologies, this method not only has higher semantic understanding capabilities, but also can provide more accurate and real-time interactive responses in complex multimodal input environments, greatly improving the practicality of the system and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 A flowchart of a metaverse-based intelligent interaction method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0080] like Figure 1 FIG. 1 is a flowchart of a metaverse-based intelligent interaction method provided by an embodiment of the present invention, wherein the method comprises the following steps:

[0081] Receiving a multimodal input signal from a user, the multimodal input signal including voice, gesture, and facial expression;

[0082] First preprocessing the multimodal input signal to generate multimodal feature data;

[0083] Obtaining user background information, and preprocessing the background information to generate background feature data;

[0084] The preprocessed multimodal feature data is fused with the background feature data to generate input data;

[0085] Perform semantic analysis on input data to obtain the user's true intention;

[0086] Dynamically adjust the interactive content and presentation method in the pre-established metaverse virtual environment based on the user's real intention;

[0087] Optimize interactive content in real time based on user feedback.

[0088] In an embodiment of the present invention, by receiving a user's multimodal input signal, including voice, gestures and facial expressions, it is possible to fully capture and understand the user's intentions, providing rich data support for subsequent interactive content generation; the method generates multimodal feature data by first preprocessing the multimodal input signal, effectively reducing the noise and redundancy of the data, and improving the efficiency and accuracy of data processing; the method also includes obtaining user background information and preprocessing it to generate background feature data. This step can ensure that the user's historical behavior, preferences and current situation can be comprehensively considered when generating input data, so that the generated interactive content is more personalized and targeted; by preprocessing the preprocessed multimodal feature data The data is fused with background feature data to generate input data, so that the input data can better reflect the user's true intention and improve the accuracy of semantic analysis; the semantic analysis step in the method can accurately capture the user's true intention by deeply analyzing the input data, and provides basic support for dynamic interaction in the metaverse virtual environment; based on the user's true intention, the interactive content and presentation method are dynamically adjusted in the pre-established metaverse virtual environment, making the interaction process more natural and smooth, enhancing the user's immersion; finally, according to the user's feedback information, the interactive content is optimized in real time to ensure that the system can be adaptively adjusted as user needs change, thereby improving the overall satisfaction of the user experience.

[0089] In summary, this method realizes adaptive dynamic adjustment of intelligent interaction in the metaverse by comprehensively utilizing multimodal input signals, background information and user feedback, significantly improves the response speed and interaction effect of the system, and has outstanding beneficial effects compared with the existing technology.

[0090] Among them, background information can come from multiple channels, including the following:

[0091] User historical behavior data: This refers to the interaction records of users during their long-term use of the system; it includes the content that the user has previously visited, the functions used, the previous search history, click records, and decisions made in similar situations. This type of information helps the system understand the user's habits and preferences and predict the user's possible needs in the current situation.

[0092] User preference data: User preference data includes various personalized settings and preferences set by users in the system, such as the user's content display preferences (such as concise or detailed), interaction method preferences (such as voice control or gesture control), topic categories of interest, areas of interest, etc. This information helps the system generate interactive content that is more in line with the user's personal preferences.

[0093] User geolocation information: Geolocation information includes the user's current location and places they have visited. This type of information is very critical in certain situations. For example, when a user wants to obtain location-related information such as weather and traffic, more targeted suggestions can be provided through the user's location information.

[0094] Device information: Device information refers to the type of device used by the user, operating system, screen resolution, network connection status, etc. This information helps adjust the way content is presented to adapt to the characteristics of the user's device and ensure the best user experience.

[0095] Virtual background information: In the metaverse, users may be in a virtual environment. Background information also includes the status and scene settings of the current virtual environment, such as the virtual world scene the user is currently in, the status of the virtual characters or objects that interact with the user, the attributes of virtual items, etc. This information helps understand the user's specific operational needs in the virtual environment.

[0096] Among them, background information provides key context for the system to understand the user's multimodal input signals, enabling the system to capture the user's true intentions more accurately.

[0097] Specifically, background information has the following advantages:

[0098] Provide contextual support: When users input voice, gestures or facial expressions, background information provides important context beyond the input signal, making it possible to associate the input signal with the user's historical behavior, current preferences, geographic location, etc., to generate more accurate interactive content.

[0099] Optimize interactive content: By combining background information, it is possible to generate interactive content that better meets the user's current needs. For example, it can provide localized recommendations based on the user's geographic location and provide customized content display based on the user's preference data.

[0100] Improve the accuracy of semantic parsing: During the semantic parsing stage, background information helps to better understand the user's true intention, especially when the user input is unclear or ambiguous, the background information can provide additional reference for semantic parsing.

[0101] For example:

[0102] Suppose a user requests to "show nearby restaurants" in the Metaverse system. The system will combine the user's geographic location information, past restaurant visit records, current virtual scene and other background information to generate a restaurant recommendation list that meets the user's needs. If the user has a record of visiting restaurants in the area, restaurants that the user has visited before or similar restaurants may be recommended first. By combining background information, the user's intentions can be understood more accurately and more appropriate recommendations can be provided.

[0103] As a preferred embodiment of the present invention, performing a first preprocessing on the multimodal input signal to generate multimodal feature data includes:

[0104] Performing a second preprocessing on the multimodal input signal to obtain initial multimodal feature data;

[0105] Perform time series analysis on the initial multimodal feature data to determine the synchronization and priority between the feature data of each modality;

[0106] Determine the weight coefficients between the feature data of each modality according to the synchronization and priority between the feature data of each modality;

[0107] According to the weight coefficients between the feature data of each modality, the feature data of each modality are fused through a fusion formula to obtain multimodal feature data.

[0108] In an embodiment of the present invention, by performing a second preprocessing on the multimodal input signal, the initial multimodal feature data can be effectively extracted, ensuring the accuracy of the subsequent time series analysis; through the time series analysis, the synchronization and priority between the feature data of each modality can be effectively determined, so that the weights can be allocated more scientifically and reasonably in the multimodal data fusion process, thereby improving the effect of data fusion; the determination of the synchronization and priority can ensure that the importance of the feature data of each modality is better reflected during data fusion, and ultimately generate more representative multimodal feature data, thereby improving the accuracy of semantic parsing and the personalization effect of the interactive content. Through this measure, complex multimodal input signals can be effectively processed, and the data processing efficiency and effect in the metaverse interaction can be improved.

[0109] As another preferred embodiment of the present invention, time series analysis is performed on the initial multimodal feature data to determine the synchronization and priority between the modal signals, including:

[0110] Decomposing the initial multimodal feature data into a state sequence for representing changes of each modal feature data at different time points;

[0111] According to all state sequences, the joint probability distribution of each modal feature data is calculated, and the joint probability distribution is used to determine the transition probability between each state sequence and the conditional probability of each modal feature data appearing under each state sequence; the calculation formula of the joint probability distribution is as follows:

[0112] ,

[0113] in, is the joint probability distribution of each modal feature data sequence corresponding to all state sequences, X is the set of each modal feature data sequence corresponding to all state sequences, is the state sequence at time t, is the characteristic data of each mode at time t, At time t=1, the initial state sequence is entered The probability of is the state sequence From the previous state sequence The transition probability obtained by the transfer, For the state sequence Below, each modal characteristic data The conditional probability of occurrence, arrive For each state in the state sequence, is the total length of the time series;

[0114] According to the joint probability distribution, the synchronization between the state sequences of the initial multimodal feature data is analyzed to determine the feature data combination;

[0115] According to the combination of feature data, combined with the transition probability between state sequences and the conditional probability of the initial multimodal feature data under each state sequence, the priority of each modal feature data is determined.

[0116] In an embodiment of the present invention, by decomposing the initial multimodal feature data into state sequences for representing the changes of each modal feature data at different time points, it is possible to accurately capture the data change trend, providing a reliable basis for the subsequent joint probability distribution calculation; by calculating the joint probability distribution of each modal feature data, the transition probability between state sequences and the conditional probability of each modal feature data can be effectively determined, so as to further analyze the synchronization between state sequences and determine the feature data combination. This analysis method can improve the accuracy of data processing, make the generated feature data combination more in line with the actual situation, and provide a scientific basis for the subsequent priority determination. Through this measure, the metaverse interaction system's ability to process multimodal data can be significantly improved, and the system's intelligence and response speed can be enhanced.

[0117] Specifically, assuming that a segment of user voice and gesture input is received and the user's facial expression is recorded at the same time, the three modal feature data are first decomposed into state sequences. For example, the voice signal is divided into voice features at several time points, the gesture signal is divided into gesture features at several time points, and the facial expression signal is divided into expression features at several time points. Then, the joint probability distribution of these state sequences is calculated to determine the synchronization between the modal features at each time point. By analyzing these synchronizations, the strong correlation between the user's voice and gesture at a certain time point and the weak correlation between expression and gesture at other time points can be identified, and finally the priority of these feature data is obtained.

[0118] in, This formula is used to calculate the joint probability distribution of the modal feature data sequences corresponding to all state sequences; the purpose of the formula is to combine the transition probabilities of the time series and the state sequence to evaluate the conditional probability of the modal feature data appearing in a specific state; first, represents the sum of all possible state sequences, ensuring that all possible state transition paths are considered; then, represents the product of the state transition probabilities from the initial state to the final state in the time series, reflecting the dynamic changes of the sequence state; then, It represents the conditional probability of the modal feature data under a given state at each time point. This joint probability distribution can be used to analyze the synchronization, priority and relationship between different states of the modal feature data, which is helpful for further modal feature fusion and optimization.

[0119] As another preferred embodiment of the present invention, according to the feature data combination, combined with the transition probability between state sequences and the conditional probability of the initial multimodal feature data under each state sequence, the priority of each modal feature data is determined, including:

[0120] According to the characteristic data combination, combined with the transition probability of each state sequence and the conditional probability of the modal characteristic data under each state sequence, the stability coefficient of each state sequence is calculated. The calculation formula is as follows:

[0121] ,in, is the stability coefficient of the state sequence, is the total length of the time series;

[0122] According to the stability coefficient of each state sequence And the conditional probability of each modal feature data under the state sequence , calculate the influence of each modal characteristic data , influence The calculation formula is as follows:

[0123] ,in, represents the influence of the i-th modal feature data, s represents all possible state sequences, To sum all possible state sequences s and integrate the influence of each state on the modal characteristic data, is the i-th modal characteristic data at time t;

[0124] According to the influence of each modal characteristic data obtained by calculation , determine the priority of each modal feature data , priority The calculation formula is as follows:

[0125] ,in, is the priority of the i-th modal feature data, is the influence of the i-th modal feature data, is the sum of the influences of all modal feature data, m is the number of modal feature data, is the influence of the j-th modal feature data.

[0126] In an embodiment of the present invention, by analyzing the combination of feature data and combining the transition probability of each state sequence and the conditional probability of the modal feature data, the stability coefficient of each state sequence can be accurately calculated, thereby improving the ability to quantify the stability of the state sequence; by combining the stability coefficient of the state sequence and the conditional probability of the modal feature data to calculate the influence of each modal feature data, the system can more accurately determine the priority of each modal feature data, ensuring that the weight of each modal feature data can be scientifically and reasonably allocated when multimodal data is fused; through this measure, the intelligence and response speed of the metaverse interactive system can be effectively improved, the system's ability to process complex multimodal data can be enhanced, and the personalization and accuracy of the user interaction experience can be improved.

[0127] Specifically, by further combining the transition probability of the state sequence and the conditional probability of the modal feature data in each state sequence, the stability coefficient of each state sequence is calculated; assuming that the system finds that the state sequence of the speech signal has the highest stability coefficient, this indicates that the speech signal is the most stable in the current situation, and then the influence of the speech signal is calculated, and it is found that its influence is also the highest, so it has the highest priority; based on this priority, the weights of each modal feature data in subsequent steps are adjusted to ensure that the speech signal occupies a dominant position in the final generation of interactive content.

[0128] in, This formula is used to calculate the stability coefficient of the state sequence , which indicates the stability of the transition between states in a given time series; the formula is obtained by calculating the state transition probability at each time point t Sum and divide by the total length T of the time series to get an average value; this average value reflects the stability of the entire state sequence, that is, the regularity and consistency of state changes; if the state transition probability is high, it means that the state sequence has good stability; conversely, frequent state changes will lead to a low stability coefficient; the stability coefficient It is crucial to analyze the performance of state sequences in multimodal interaction systems and helps to optimize the synchronization and priority setting of modal features.

[0129] in, This formula is used to calculate the modal characteristic data Influence ; The formula is obtained by taking the stability coefficient of the state sequence and the conditional probability of the modal characteristic data appearing in this state sequence Multiply and sum all possible state sequences to obtain the influence of the modal characteristic data on the overall system; influence The larger the value, the more important the modal feature data is in the multimodal interaction system. Through this calculation, the system can identify the modal features that contribute most to the user's intention judgment, thereby optimizing the data processing and decision-making process.

[0130] in, This formula is used to determine the priority of modal feature data ; By taking the influence of a certain modal feature data Normalized with the total influence of all modal feature data, the formula derives the relative priority of the modal feature data; Indicates the importance of the modal feature data relative to other data in the multimodal fusion process; this priority plays a key role in multimodal data fusion and final decision-making, helping the system to prioritize the features that have the greatest impact on user intent and optimize the interactive experience.

[0131] As another preferred embodiment of the present invention, semantic analysis is performed on the input data to obtain the user's real intention, including:

[0132] Through the convolutional neural network, the input data is used to extract speech features, gesture features and facial expression features;

[0133] Arrange the speech features, gesture features and facial expression features respectively according to the time series order to obtain a speech feature matrix, a gesture feature matrix and a facial expression feature matrix;

[0134] According to the speech feature matrix, gesture feature matrix and facial expression feature matrix, a multi-layer semantic network structure is constructed;

[0135] Through the cross-modal alignment formula, semantic mapping between different modal features is achieved to obtain a semantic feature matrix; the cross-modal alignment formula is as follows:

[0136] ,

[0137] in, is the aligned semantic feature matrix, is the speech feature matrix, is the gesture feature matrix The transpose of is the facial expression feature matrix, d is the dimension of the feature vector, is the normalization function;

[0138] According to the semantic feature matrix and background feature data, the user's true intention is calculated; the output of the user's true intention is realized through the following calculation formula:

[0139] ,

[0140] in, The output of the user's true intention. is the background feature data, is the semantic feature matrix, is the weighted coefficient between the background feature data and the semantic feature matrix, is the normalization function.

[0141] In an embodiment of the present invention, by constructing a multi-layer semantic network structure, the multimodal feature semantic information of speech, gesture and facial expression can be extracted respectively, and the cross-modal alignment of the semantic mapping between multimodal features is realized, ensuring that the feature data of each modality can cooperate with each other in the semantic parsing process, thereby improving the accuracy of semantic understanding; through the use of the cross-modal alignment formula, the feature data of speech, gesture and facial expression can be effectively aligned, and a more accurate semantic feature matrix can be generated, and finally the user's true intention can be calculated. Through this measure, the intelligence level of the interactive content in the metaverse virtual environment can be significantly improved, so that the system can better understand the user's multimodal input and enhance the immersiveness of the user experience.

[0142] Specifically, assuming that the user is interacting with a virtual assistant in the metaverse, the user's semantic intention needs to be analyzed; by constructing a multi-layer semantic network structure, the semantic information of the user's voice, gestures, and facial expressions are extracted respectively; voice information may indicate that the user is asking about the weather, gesture information may indicate that the user is pointing in a certain direction, and facial expressions may indicate that the user is confused; using the cross-modal alignment formula, these modal features are aligned to generate an aligned semantic feature matrix; by combining the user's background feature data, such as the user's previous behavior records, the user's true intention is finally calculated, that is, the user may be looking for weather information in a certain direction.

[0143] Among them, the voice feature matrix, gesture feature matrix, and facial expression feature matrix can all be established by obtaining multimodal feature data, which belongs to the existing technology and will not be described in detail.

[0144] in, This formula is used to achieve semantic alignment between features of different modalities to generate a cross-modal semantic feature matrix ; The formula is to transform the speech feature matrix Transpose of the gesture feature matrix Multiply and then divide by the square root of the feature vector dimension d for normalization; this normalization step can prevent the influence of the feature vector length on the calculation result; then convert the result into a probability distribution through the Softmax function, and finally multiply it by the facial expression feature matrix , generate the aligned semantic feature matrix ; This matrix is ​​used to unify semantic expressions in multimodal interactions and improve the system's ability to understand multimodal input.

[0145] in, This formula is used to calculate the user's true intention ; By combining background feature data And the aligned semantic feature matrix , and use the weighting coefficient By balancing the two, the formula can derive the user's true intent under the current context and historical behavior; the Softmax function is used to convert the result into a probability distribution to determine the most likely user intent; in this way, the system can more accurately judge the user's needs and expectations, and make corresponding interaction adjustments.

[0146] As another preferred embodiment of the present invention, according to the real intention of the user, the interactive content and presentation method are dynamically adjusted in the pre-established metaverse virtual environment, including:

[0147] Generate preliminary interactive content based on the user's real intention;

[0148] The preliminary interactive content is scored according to the scoring formula to obtain an interactive content scoring set; the calculation formula for the interactive content scoring is:

[0149] ,

[0150] in, For the initial interactive content rating, is the influence of the i-th characteristic factor on user u and content c, is the weight of the i-th feature factor, and n is the number of feature factors.

[0151] In an embodiment of the present invention, by acquiring the user's historical behavior data, preference data and current environmental information, preliminary interaction content can be accurately generated and scored using a scoring formula, so that the preliminary interaction content can better reflect the user's needs and preferences, thereby enhancing the system's response speed and interaction effect.

[0152] Specifically, based on the user's true intention, several preliminary interactive contents were generated in the metaverse virtual environment, such as several possible weather forecast display methods; the user's historical behavior data (for example, the user likes a simple display method), preference data (for example, the user prefers to use charts to display weather information), and current environmental information (for example, the user is currently in a specific location) were obtained; then, the preliminary interactive contents were scored, and it was found that the simple chart display method scored the highest; then, the emotional influencing factor was used for adjustment. Assuming that the user is currently showing fatigue, the priority of complex display methods may be further reduced, and finally the interactive content of the simple chart is determined.

[0153] The characteristic factors include the following:

[0154] User preference characteristic factors: Used to measure the degree of match between recommended content and user historical preferences; user preferences can be expressed by the user's browsing history, click records, likes or favorites, etc., which can reflect whether the content meets the user's long-term interests and needs.

[0155] Content-related feature factors: Used to reflect the characteristics of the content itself, such as the type, theme, length, format, etc. of the content; used to evaluate the fit between the content and the user's current needs; for example, for short video platforms, the length of the content may be an important content feature factor.

[0156] Situational factors: Used to reflect the user's current contextual information, such as the user's current geographic location, time, device type, network connection status, etc.; contextual feature factors help the system recommend suitable content in specific contexts, such as recommending short-duration content when the user is commuting and recommending long-duration content when the user is resting.

[0157] User behavior characteristic factors: Used to measure the correlation between the user's current behavior and the recommended content; it can be calculated based on the user's click behavior, dwell time, interaction frequency, etc., and used to predict the user's interest in the current recommended content.

[0158] Social characteristics factor: Reflects the impact of a user's social network relationships or group behaviors on content recommendations; for example, the content that a user's friends or followers have recently interacted with may have an impact on the user's content recommendations.

[0159] In summary, for different feature factors, the existing technology is used to reflect the influence of the i-th feature factor on user u and content c, and then different weights are assigned to each feature factor. , the system can comprehensively consider various factors and come up with a preliminary interactive content score that better meets user needs .

[0160] As another preferred embodiment of the present invention, according to the real intention of the user, the interactive content and presentation method are dynamically adjusted in the pre-established metaverse virtual environment, further comprising:

[0161] Obtain user behavior history scores, content relevance scores, and emotional state scores;

[0162] The interactive content score set is sorted by multi-dimensional priority according to the user's feedback information to obtain an interactive content priority score set; the calculation formula for the interactive content priority is as follows:

[0163] ,

[0164] in, Score the priority of content c for user u, Score user behavior history, Score the user's emotional state, Score the relevance of the content. Rate the situation. is the feedback correction coefficient, They are the weight coefficients of the behavior history score and the affective state score, the content relevance score, the situation score, and the feedback correction coefficient;

[0165] Generate interactive content based on the results of multi-dimensional priority sorting.

[0166] In an embodiment of the present invention, by obtaining the user behavior history score, content relevance score and emotional state score, the metaverse-based intelligent interaction method can accurately reflect the user's historical behavior, content preference and current emotional state, and provide a reliable data basis for the multi-dimensional priority sorting of interactive content; by multi-dimensional priority sorting of the interactive content collection, it can ensure that the generated interactive content is more in line with the user's needs and current situation, and improve the personalization and accuracy of the user experience; through this measure, the intelligence and response speed of the metaverse interactive system can be significantly improved, and the system's interactive effect and user satisfaction can be enhanced.

[0167] Specifically, after determining the interactive content, multi-dimensional priority sorting is performed based on the user's behavior history score, content relevance score, and emotional state score; assuming that the user has frequently checked the weather in a certain area before (high behavior history score), the current displayed content is related to the user's area of ​​interest (high content relevance score), and the user's emotional state is relaxed (moderate emotional state score); through multi-dimensional priority sorting, the weather information content to be displayed is ultimately determined; at the same time, context and feedback information will also affect the final display results, and the final priority order will be adjusted based on these factors to generate the content that the user ultimately sees.

[0168] Among them, the user behavior history score, content relevance score, and emotional state score can all be calculated using the above scoring formula get.

[0169] in, This formula is used to calculate the priority score of user u for content c ; The formula combines the user behavior history score , emotional state score , content relevance score , Situational Rating and feedback correction factor , and the scores were weighted averaged; the weight coefficient They correspond to user behavior, content relevance and contextual influence respectively; through this priority score, the system can better determine which content should be recommended to users first, thereby improving the user's interactive experience.

[0170] Among them, user behavior history score , emotional state score , content relevance score and contextual ratings All can be done through the above The calculation formula is used to calculate it.

[0171] As another preferred embodiment of the present invention, the interactive content is optimized in real time according to the user's feedback information, including:

[0172] Obtaining real-time feedback information from users and performing a third preprocessing on the information to obtain real-time feedback data;

[0173] Fusion of real-time feedback data and historical feedback data to generate a multi-dimensional feedback feature matrix;

[0174] The multi-dimensional feedback feature matrix is ​​subjected to a fourth preprocessing, and the parameters for generating the interactive content are adjusted to obtain the adjustment parameters; the calculation formula of the adjustment parameters is:

[0175] ,

[0176] in, To adjust the parameters for generating interactive content, Parameters for generating interactive content, is the learning rate, which determines the step size of each parameter update. is the loss function based on feedback data About parameters The gradient of To balance the coefficient, adjust the influence weight of different loss functions in the final parameter update. is the loss function based on the metaverse virtual environment structure About parameters gradient.

[0177] In an embodiment of the present invention, by acquiring the user's real-time feedback information and converting it into feedback data, and then fusing it with the historical feedback data, a multi-dimensional feedback feature matrix can be generated, providing data support for the real-time optimization of the interactive content; by preprocessing the multi-dimensional feedback feature matrix based on a decomposition algorithm, the parameters for generating the interactive content can be dynamically updated to ensure that the interactive content can be adjusted in real time as user needs change, thereby improving the overall satisfaction of the user experience; through this measure, the response speed and intelligence of the Metaverse interactive system can be significantly improved, and the interactive effect and user satisfaction of the system can be enhanced.

[0178] Specifically, after the final interactive content is generated and displayed, the user feedback information is continuously monitored; assuming that the user is satisfied with the displayed weather chart and does not conduct any additional interaction (real-time feedback information); the feedback data is integrated with the user's historical feedback data to generate a multi-dimensional feedback feature matrix; if it is found that the user is satisfied with similar chart displays many times, the parameters for generating interactive content will be adjusted to increase the priority of simple chart displays, so as to better meet user needs when displaying similar information in the future.

[0179] in, This formula is used to generate parameter updates for interactive content; the formula combines the loss function based on feedback data And the loss function based on the metaverse virtual environment structure Gradient information, adjust the current parameters , generate new parameters ; Learning rate Control the step size of each update, balance coefficient Used to adjust the weights of different loss functions in the final update; through this update mechanism, the system can dynamically adjust the generation process of interactive content based on user feedback, thereby improving the system's adaptability and response speed.

[0180] Among them, the definition is: It is a loss function based on user feedback, which is used to measure the gap between the output of the virtual environment and the user's expectations; specifically, the user will provide feedback after interacting with the system, and the system uses this feedback to adjust the parameters of the virtual environment to reduce errors in future predictions.

[0181] Calculation method: The loss function can be calculated using data such as ratings, click behavior, and browsing time given by the user. For example, if the content recommended by the system fails to receive positive feedback from the user, the value of the loss function will increase, thereby prompting the virtual environment to optimize the recommendation results during the next adjustment. Common calculation methods include mean square error or cross entropy loss.

[0182] effect: It plays a direct optimization role in the virtual environment training process, enabling the virtual environment to better respond to user needs and gradually improve the system's personalized recommendation capabilities and user satisfaction.

[0183] Among them, the definition is: It is a loss function based on the structure of the virtual environment of the metaverse, which is mainly used to maintain and improve the structure and generalization ability of the virtual environment; Optimize based on the internal characteristics of the virtual environment.

[0184] Calculation method: This loss function is usually implemented through regularization methods, such as L2 regularization (also known as weight decay), to avoid overfitting of the virtual environment. The main purpose is to limit the growth of virtual environment parameters and ensure that the virtual environment has better generalization ability when processing new data.

[0185] As another preferred embodiment of the present invention, the interactive content is optimized in real time according to the user's feedback information, further comprising:

[0186] Feedback information is divided into three categories: immediate feedback, periodic feedback and long-term feedback;

[0187] According to the classification results of the feedback data, the interactive content is adjusted in multiple dimensions. The calculation formula for the multi-dimensional weight adjustment is:

[0188] ,in, is the adjusted weight, is the impact weight of the i-th type of feedback data, is the weight coefficient of each type of feedback data.

[0189] In an embodiment of the present invention, by dividing feedback information into immediate feedback, periodic feedback and long-term feedback, the intelligent interaction method based on the metaverse can classify and process different types of feedback information, providing a scientific basis for the multi-dimensional weight adjustment of the interactive content; by performing multi-dimensional weight adjustment on the interactive content, it can ensure that the system responds to different types of feedback information more accurately and efficiently, thereby generating interactive content that better meets user needs; through this measure, the response speed and intelligence of the metaverse interactive system can be significantly improved, and the interactive effect and user satisfaction of the system can be enhanced.

[0190] Specifically, during the continuous operation process, different types of feedback information are received from users, including instant feedback (users' immediate reaction to the currently displayed content), periodic feedback (users' interactive performance over a period of time) and long-term feedback (users' usage behavior over a long period of time); based on the classification results of these feedback information, the interactive content is adjusted in multiple dimensions; for example, it may be found that instant feedback is more important, so when generating interactive content, the impact of instant feedback information will be given priority, and the corresponding weight coefficient will be adjusted to ensure that user needs can be responded to more quickly and content that is more in line with user expectations can be generated.

[0191] in, This formula is used to adjust the weights of different types of feedback information; the formula is based on the weights of immediate feedback, periodic feedback, and long-term feedback. Perform weighted summation to obtain the adjusted weight ; Weight coefficient Indicates the relative importance of each type of feedback information; by weighting different types of feedback, the system can more accurately evaluate and respond to user feedback, thereby improving the quality of interactive content and user satisfaction.

[0192] An embodiment of the present invention further provides a metaverse-based intelligent interaction system, which is applied to any of the above methods, and the system includes:

[0193] A multimodal input module, used to receive a multimodal input signal from a user, the multimodal input signal including voice, gestures and facial expressions;

[0194] A multimodal input signal processing module, used for first preprocessing the multimodal input signal to generate multimodal feature data;

[0195] A background information processing module is used to obtain the user's background information, pre-process the background information, and generate background feature data;

[0196] A data fusion module is used to fuse the pre-processed multi-modal feature data with the background feature data to generate input data;

[0197] The semantic understanding module is used to perform semantic analysis on the input data to obtain the user's true intention;

[0198] A dynamic adjustment module is used to dynamically adjust the interactive content and presentation method in the pre-established metaverse virtual environment according to the user's real intention;

[0199] The feedback optimization module is used to optimize the interactive content in real time according to the user's feedback information.

[0200] It should be noted that the system is a system corresponding to the above method, and all implementation methods in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0201] The embodiment of the present invention further provides a computing device, including: a processor, a memory storing a computer program, and when the computer program is executed by the processor, the method described above is executed. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0202] The embodiment of the present invention further provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute the above method. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0203] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An intelligent interaction method based on the metaverse, characterized in that: The method comprises the following steps: Receiving a multimodal input signal from a user, and first preprocessing the multimodal input signal to generate multimodal feature data, wherein the multimodal input signal includes voice, gesture, and facial expression; Obtaining user background information, and preprocessing the background information to generate background feature data; The preprocessed multimodal feature data is fused with the background feature data to generate input data; Through the convolutional neural network, the input data is used to extract speech features, gesture features and facial expression features; Arrange the speech features, gesture features and facial expression features respectively according to the time series order to obtain a speech feature matrix, a gesture feature matrix and a facial expression feature matrix; According to the speech feature matrix, gesture feature matrix and facial expression feature matrix, a multi-layer semantic network structure is constructed; Through the cross-modal alignment formula, semantic mapping between different modal features is achieved to obtain a semantic feature matrix; According to the semantic feature matrix and background feature data, the user's true intention is calculated; Dynamically adjust the interactive content and presentation method in the pre-established metaverse virtual environment based on the user's real intention; Optimize interactive content in real time based on user feedback; Performing a first preprocessing on the multimodal input signal to generate multimodal feature data includes: Performing a second preprocessing on the multimodal input signal to obtain initial multimodal feature data; Perform time series analysis on the initial multimodal feature data to determine the synchronization and priority between the feature data of each modality; Determine the weight coefficients between the feature data of each modality according to the synchronization and priority between the feature data of each modality; According to the weight coefficients between the feature data of each modality, the feature data of each modality are fused through a fusion formula to obtain multi-modal feature data; Perform time series analysis on the initial multimodal feature data to determine the synchronization and priority between the modal signals, including: Decomposing the initial multimodal feature data into a state sequence for representing changes of each modal feature data at different time points; According to all state sequences, the joint probability distribution of each modal feature data is calculated, and the joint probability distribution is used to determine the transition probability between each state sequence and the conditional probability of each modal feature data appearing under each state sequence; the calculation formula of the joint probability distribution is as follows: ， in, is the joint probability distribution of each modal feature data sequence corresponding to all state sequences, X is the set of each modal feature data sequence corresponding to all state sequences, is the state sequence at time t, is the characteristic data of each mode at time t, At time t=1, the initial state sequence is entered The probability of is the state sequence From the previous state sequence The transition probability obtained by the transfer, For the state sequence The characteristic data of each mode are The conditional probability of occurrence, arrive are each state in the state sequence, T is the total length of the time series; According to the joint probability distribution, the synchronization between the state sequences of the initial multimodal feature data is analyzed to determine the feature data combination; According to the feature data combination, the priority of each modal feature data is determined in combination with the transition probability between state sequences and the conditional probability of the initial multimodal feature data under each state sequence; According to the combination of feature data, combined with the transition probability between state sequences and the conditional probability of the initial multimodal feature data under each state sequence, the priority of each modal feature data is determined, including: According to the combination of characteristic data, combined with the transition probability of each state sequence and the conditional probability of the modal characteristic data under each state sequence, the stability coefficient of each state sequence is calculated. The calculation formula is as follows: ,in, is the stability coefficient of the state sequence; According to the stability coefficient of each state sequence And the conditional probability of each modal feature data under the state sequence , calculate the influence of each modal characteristic data , influence The calculation formula is as follows: ,in, represents the influence of the i-th modal feature data, s represents all possible state sequences, To sum all possible state sequences s and integrate the influence of each state on the modal characteristic data, is the i-th modal characteristic data at time t; According to the influence of each modal characteristic data obtained by calculation , determine the priority of each modal feature data , priority The calculation formula is as follows: ,in, is the priority of the i-th modal feature data, is the influence of the i-th modal feature data, is the sum of the influences of all modal feature data, m is the number of modal feature data, is the influence of the j-th modal feature data.

2. According to claim 1, a metaverse-based intelligent interaction method is characterized in that: The cross-modal alignment formula is as follows: ， in, is the aligned semantic feature matrix, is the speech feature matrix, is the gesture feature matrix The transpose of is the facial expression feature matrix, d is the dimension of the feature vector, is the normalization function; The output of the user's true intention is achieved through the following calculation formula: ， in, The output of the user's true intention. is the background feature data, is the semantic feature matrix, is the weighting coefficient between the background feature data and the semantic feature matrix.

3. The intelligent interaction method based on the metaverse according to claim 2, characterized in that: Based on the user's real intention, the interactive content and presentation method are dynamically adjusted in the pre-established metaverse virtual environment, including: Generate preliminary interactive content based on the user's real intention; The preliminary interactive content is scored according to the scoring formula to obtain an interactive content scoring set; the calculation formula for the interactive content scoring is: ， in, For the initial interactive content rating, is the influence of the i-th characteristic factor on user u and content c, is the weight of the i-th feature factor, n is the number of feature factors; the feature factors include user preference feature factors, content-related feature factors, situational feature factors, user behavior feature factors and social feature factors.

4. The intelligent interaction method based on the metaverse according to claim 3, characterized in that: Based on the user's real intention, the interactive content and presentation method are dynamically adjusted in the pre-established metaverse virtual environment, including: Obtain user behavior history scores, content relevance scores, emotional state scores, and real-time scenario scores; The interactive content score set is sorted by multi-dimensional priority according to the user's feedback information to obtain an interactive content priority score set; the calculation formula for the interactive content priority is as follows: ， in, Score the priority of content c for user u, Score user behavior history, Score the user's emotional state, Score the relevance of the content. Rate the situation. is the feedback correction coefficient, They are the weight coefficients of the behavior history score and the affective state score, the content relevance score, the situation score, and the feedback correction coefficient; Generate interactive content based on the interactive content priority score set.

5. The intelligent interaction method based on the metaverse according to claim 4, characterized in that: Based on user feedback, interactive content is optimized in real time, including: Obtaining user feedback information and performing a third preprocessing on it to obtain feedback data; Fuse the feedback data with historical feedback data to generate a multi-dimensional feedback feature matrix; The multi-dimensional feedback feature matrix is ​​subjected to a fourth preprocessing, and the parameters for generating the interactive content are adjusted to obtain the adjustment parameters; the calculation formula of the adjustment parameters is: ， in, To adjust the parameters for generating interactive content, Parameters for generating interactive content, is the learning rate, which determines the step size of each parameter update. is the loss function based on feedback data About parameters The gradient of To balance the coefficient, adjust the influence weight of different loss functions in the final parameter update. is the loss function based on the metaverse virtual environment structure About parameters gradient.

6. The intelligent interaction method based on the metaverse according to claim 5, characterized in that: Based on user feedback, interactive content is optimized in real time, including: The feedback data is divided into three categories: immediate feedback, periodic feedback and long-term feedback; According to the classification results of the feedback data, the interactive content is adjusted in multiple dimensions. The calculation formula for the multi-dimensional weight adjustment is: ,in, is the adjusted weight, is the impact weight of the i-th type of feedback data, is the weight coefficient of each type of feedback data.

7. An intelligent interactive system based on the metaverse, applied to any of the methods described in claims 1-6, characterized in that: The system comprises: A multimodal information processing module, used to receive a multimodal input signal from a user, and first pre-process the multimodal input signal to generate multimodal feature data, wherein the multimodal input signal includes voice, gesture and facial expression; A background information processing module is used to obtain the user's background information, pre-process the background information, and generate background feature data; A data fusion module is used to fuse the pre-processed multi-modal feature data with the background feature data to generate input data; The feature extraction module is used to extract speech features, gesture features and facial expression features from the input data through a convolutional neural network; A feature matrix building module is used to arrange the speech features, gesture features and facial expression features according to the time series order to obtain a speech feature matrix, a gesture feature matrix and a facial expression feature matrix; A multi-layer semantic network structure building module is used to build a multi-layer semantic network structure according to a speech feature matrix, a gesture feature matrix and a facial expression feature matrix; A semantic feature matrix generation module is used to achieve semantic mapping between different modal features through a cross-modal alignment formula to obtain a semantic feature matrix; The real intention generation module is used to calculate the user's real intention based on the semantic feature matrix and background feature data; A dynamic adjustment module is used to dynamically adjust the interactive content and presentation method in the pre-established metaverse virtual environment according to the user's real intention; The feedback optimization module is used to optimize the interactive content in real time according to the user's feedback information.

Images