VR virtual character behavior generation method and system based on multi-modal data fusion

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a VR virtual character behavior generation method and system based on multi-modal data fusion. The system comprises a virtual character generation center, a multi-modal acquisition module, a behavior generation and matching module, a behavior fusion module, a behavior vividness analysis module, a naturalness analysis module, a behavior generation improvement module and a visual feedback module. Through cooperative work of the modules, the purposes of accurate interaction, high immersion and smooth experience are finally achieved, that is, multi-modal data collection ensures that the intention of a user is comprehensively understood, behavior matching ensures that instruction response is accurate, meanwhile, consistency of virtual role behaviors and the intention of the user is guaranteed in the process, and the user experience is improved. According to performance evaluation, interaction obstacles are eliminated from three dimensions of fitness, fidelity and naturalness, so that a user can smoothly express requirements in a VR environment and naturally interact with a virtual character.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for generating VR virtual character behavior based on multimodal data fusion. Background Technology

[0002] With the rapid development of technology, virtual reality (VR) technology has made significant progress in the past few years and has been widely used in many fields such as games, education, medical care, and industrial design, bringing users an immersive experience. In VR applications, the generation of virtual character behavior is a crucial link, which directly affects the user's sense of immersion and interactive experience. A virtual character that can naturally and realistically express various behaviors can make users more engaged in the virtual environment, enhancing the attractiveness and practicality of VR applications.

[0003] However, current methods for generating VR virtual character behavior based on multimodal data fusion still face some challenges. On the one hand, there is a lack of an effective verification mechanism for behavior matching results. When a virtual character exhibits mismatched behavior (such as the user commanding "turn left" but the virtual character turning "right"), the system cannot identify the root cause of the problem in a timely manner and can only rely on subjective user feedback for adjustments, leading to continuous interaction errors that affect the experience. On the other hand, current evaluations of the quality of VR virtual character behavior focus primarily on the basic level of whether it responds to commands, ignoring key dimensions such as whether the behavior is realistic and whether the movements are natural. This results in highly subjective evaluation results with low reference value, and also makes it impossible to accurately pinpoint areas for improvement, leading to slow technological iteration.

[0004] Therefore, a solution is proposed. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for generating VR virtual character behavior based on multimodal data fusion, in order to solve the aforementioned technical defects.

[0006] The objective of this invention can be achieved through the following technical solution: a VR virtual character behavior generation system based on multimodal data fusion, including a virtual character generation center, a multimodal acquisition module, a behavior generation and matching module, a behavior fusion module, a behavior realism analysis module, a naturalness analysis module, a behavior generation improvement module, and a visual feedback module; The multimodal acquisition module collects multidimensional data from users through multiple sensors and sends it to the virtual character generation center for storage. The behavior generation and matching module is used to process and analyze the multidimensional data of retrieved users to obtain an input-behavior correspondence table. The behavior fusion module is used to analyze the behavior fit of user input data and character behavior data collected during the interaction between the user and the VR virtual character, and to perform discriminant analysis on the obtained objective matching ratio to obtain stable signals or interference signals. The stable signal-based behavior realism analysis module is used to analyze the behavior realism of collected user behavior data and VR virtual character data to obtain qualified or unqualified signals. The naturalness analysis module is used to analyze the naturalness of each action of the collected VR virtual character to obtain qualified or unqualified signals. The behavior generation improvement module is used to integrate, interact with, and summarize the output results of behavior realism acquisition analysis and behavior naturalness acquisition analysis to obtain a list of improvements for VR virtual character generation.

[0007] Preferably, the analysis process of the behavior generation and matching module is as follows: S1: Preprocess and fuse the collected multidimensional data from users to obtain multimodal fusion data. Preprocessing includes cleaning and standardization. S2: Generates behavioral decisions for virtual characters based on multimodal fusion data and a pre-set behavior generation model. Behavioral decisions include action commands and facial expression commands. S3: Organize user input methods and specific commands into a list; S4: Based on the list, bind a unique correct role behavior to each input instruction to form an input-behavior mapping table.

[0008] Preferably, the analysis process of the behavior fusion module is as follows: Acquire user input data and character behavior data during the interaction between the user and the VR virtual character; User input data and character behavior data within the same interaction cycle are paired using timestamps. An interaction cycle is defined as the period from the start of user input to the end of VR virtual character behavior. n sets of user-VR virtual character interaction cycles are obtained, where n is a natural number greater than zero. The user input data and character behavior data are checked and paired one by one according to the input-behavior mapping table to obtain the pairing results. The pairing results include correct matches and incorrect matches. If the behavior of the VR virtual character matches the input-behavior correspondence table, it is judged as a correct match; if the behavior of the VR virtual character does not match the input-behavior correspondence table, it is judged as an incorrect match.

[0009] Preferably, based on the pairing results, the number of correct matches is set to N1, the ratio between N1 and the total number of interactions is set as the objective matching ratio, and the objective matching ratio is judged to see if it exceeds the preset objective matching ratio threshold. If it exceeds the threshold, a stable signal is generated; if it does not exceed the threshold, an interference signal is generated.

[0010] Preferably, the analysis process of the behavior realism analysis module is as follows: Collect user behavior data and VR virtual character data, and perform data format unification and noise removal on the collected user behavior data and VR virtual character data; By evaluating the action posture and facial expression and voice of the processed user behavior data and VR virtual character data, action similarity and facial expression and voice similarity are obtained. The weight coefficients W1 and W2 corresponding to action similarity and facial expression-voice similarity are retrieved. Both W1 and W2 are greater than zero. The behavior realism is calculated based on action similarity × W1 + facial expression-voice similarity × W2. The system determines whether the behavior realism reaches a preset behavior realism threshold, and obtains a qualified signal or an unqualified signal.

[0011] Preferably, the analysis process of the naturalness analysis module is as follows: SS1: Collect the time periods of each action of the VR virtual character (the duration between the start time and the end time of the action), divide the action time period into i time points, where i is a natural number greater than zero, and obtain the skeletal key point position data of the VR virtual character in each time period. SS2: Based on the position difference and time interval between two adjacent time points, the instantaneous velocity at each time point is obtained using the formula velocity = position difference / time interval, generating a time-instantaneous velocity change curve; SS3: Then, based on the velocity difference and time interval between two adjacent time points, use the formula acceleration = velocity difference / time interval to obtain the instantaneous acceleration at each time point and generate a time-acceleration change curve; SS4: Obtain the maximum value of the variance between the time-instantaneous velocity change curve and the time-acceleration change curve, and set it as the smoothness index value; The smoothness index value is processed to determine whether the behavior is non-smooth or smooth. Obtain the percentage of behaviors corresponding to behavior smoothing, set the percentage of behaviors corresponding to behavior smoothing as the behavior naturalness, and perform discrimination processing on the behavior naturalness to obtain the non-compliant signal or compliant signal.

[0012] Preferably, the analysis process of the behavior generation improvement module is as follows: Based on the interactive analysis of qualified signals, unqualified signals, qualified signals, and unqualified signals, output signals or updated signals are obtained; When an update signal is generated, the analysis results of the improved realism and naturalness of the VR virtual character's behavior are obtained until the output signal is generated. When the output signal is generated, information such as the number of times the VR virtual character has been improved and the direction of improvement is obtained; An improvement list for VR virtual characters is generated based on the number of improvements and the direction of improvement.

[0013] The beneficial effects of this invention are as follows: This invention provides data support for a comprehensive understanding of user behavior by complementing different types of data. Through standardized data processing and behavior binding processes, it helps to ensure the consistency between virtual character behavior and user intent from a process perspective, reducing interaction errors caused by behavior mismatch. Furthermore, it allows for real-time evaluation from the perspective of objective matching ratio, enabling developers or users to promptly grasp the matching status between virtual character behavior and user input, and preventing invalid interactions from continuously affecting the user experience.

[0014] This invention further ensures the rationality and naturalness of the generated behavior in the VR environment. From a visual perspective, it evaluates whether the virtual character's movements are smooth and coordinated, and whether they conform to human movement habits and behavioral logic. Through these performance evaluations, the behavior generation model is continuously optimized, ultimately achieving the goal of precise interaction, high immersion, and smooth experience. Attached Figure Description

[0015] The invention will now be further described with reference to the accompanying drawings; Figure 1 This is a flowchart of the system of the present invention; Figure 2 This is a partial reference diagram of Embodiment 1 of the present invention; Figure 3 This is a reference diagram of the method of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments; Example 1: Please refer to Figures 1 to 3As shown, the present invention is a VR virtual character behavior generation system based on multimodal data fusion, including a virtual character generation center, a multimodal acquisition module, a behavior generation and matching module, a behavior fusion module, a behavior realism analysis module, a naturalness analysis module, a behavior generation improvement module, and a visual feedback module. The virtual character generation center has bidirectional communication connections with the multimodal acquisition module and the behavior generation and matching module. The virtual character generation center has a one-way communication connection with the behavior fusion module. The behavior fusion module has a one-way communication connection with the behavior realism analysis module and the naturalness analysis module. The behavior realism analysis module and the naturalness analysis module have a one-way communication connection with the behavior generation improvement module. The naturalness analysis module has a one-way communication connection with the visual feedback module. The behavior generation improvement module has a one-way communication connection with the virtual character generation center. The multimodal acquisition module collects multidimensional data from users through multiple sensors (such as visual sensors, auditory sensors, etc.) and sends it to the virtual character generation center for storage. The multidimensional data includes visual data, auditory data, behavioral data, etc. Visual data includes information such as user gestures, facial expressions, and objects in the virtual environment, collected using devices such as high-definition cameras and depth cameras; Auditory data includes information such as user voice commands and ambient sounds collected through a microphone; Among them, the microphone can capture sounds in different frequency ranges. For users' voice commands, it can accurately identify their content and emotional tendencies. For example, in VR tour guide applications, users can ask for information about attractions by voice, and the system can respond accurately according to the voice commands. For environmental sounds, such as wind and rain, it can enhance the realism of the virtual environment. Behavioral data includes information such as pressure and vibration of objects touched by users when interacting with the virtual environment, collected using devices such as tactile sensors and tactile gloves; Through these multimodal sensors, we can collect a rich variety of data. Visual data is intuitive and information-rich, and can present users' behavior, posture, and facial expressions. Auditory data conveys users' intentions and environmental information in the form of speech and sound. Behavioral data focuses on perceiving the details of the physical interaction between users and the virtual environment. These different types of data complement each other, providing the possibility for a comprehensive understanding of users' behavior. The behavior generation and matching module is used to process and analyze the multidimensional user data. The specific data processing and behavior matching analysis process is as follows: S1: Preprocess and fuse the collected multidimensional data from users to obtain multimodal fusion data. Preprocessing includes cleaning, standardization, etc. S2: Generates behavioral decisions for virtual characters based on multimodal fusion data and a pre-set behavior generation model. Behavioral decisions include action commands, facial expression commands, etc. S3: Organize the user's input methods (such as voice, gestures, etc.) and specific instructions (such as walk forward, turn left, etc.) into a list; S4: Based on the list, bind a unique and correct character behavior to each input instruction to form an input-behavior mapping table, and send the obtained input-behavior mapping table to the virtual character generation center for storage.

[0018] Example 2: Evaluate the interaction effects to see if the virtual character's behavior can accurately respond to the user's intentions and interact reasonably with other elements in the VR environment, thereby improving the quality of the VR virtual character's behavior and the user experience; The behavior fusion module is used to analyze the behavior fit of user input data and character behavior data collected during the interaction between the user and the VR virtual character. The specific behavior fit analysis process is as follows: Acquire user input data (such as recording input type (e.g., voice), input content (e.g., voice text, gesture coordinates)) and character behavior data (e.g., behavior start / end timestamps, behavior parameters (e.g., walking speed, interface pop-up delay)) during the user's interaction with the VR virtual character. User input data and character behavior data within the same interaction cycle are paired using timestamps. An interaction cycle is defined as the period from the start of user input to the end of VR virtual character behavior. n sets of user-VR virtual character interaction cycles are obtained, where n is a natural number greater than zero. The user input data and character behavior data are checked and paired one by one according to the input-behavior mapping table to obtain the pairing results. The pairing results include correct matches and incorrect matches. If the behavior of the VR virtual character matches the input-behavior correspondence table, it is judged as a correct match; if the behavior of the VR virtual character does not match the input-behavior correspondence table, it is judged as an incorrect match. Based on the pairing results, the number of correct matches is set as N1, and the ratio between N1 and the total number of interactions is set as the objective matching ratio. The objective matching ratio is judged to see if it exceeds the preset objective matching ratio threshold. If it exceeds the threshold, a stable signal is generated; if it does not exceed the threshold, an interference signal is generated. The visual feedback module is used to respond to stable or interference signals and immediately display the warning text "Stable Behavior Match" corresponding to the stable signal or the warning text "Inefficient Behavior Match" corresponding to the interference signal, so as to intuitively understand the matching status between user behavior and VR virtual character behavior, which helps to provide timely feedback on the effectiveness of the behavior generated by VR virtual character. When generating a stable signal, the behavior realism analysis module is used to perform behavior realism acquisition and analysis on the collected user behavior data and VR virtual character data. The specific behavior realism acquisition and analysis process is as follows: Collect user behavior data: Record human actions (walking, interaction, etc.) in different scenarios through motion capture devices (such as OptiTrack, Kinect), and simultaneously collect facial expression data (using facial capture devices such as Faceware) and voice tone data to generate a labeled real behavior benchmark library; Collect VR virtual character data: Record the behavior output of virtual characters in the same scene, including skeletal key point coordinates, facial feature vectors, motion speed / acceleration curves, and multimodal input (voice, gestures) during user interaction; The collected user behavior data and VR virtual character data are processed to unify the data format (such as standardizing the skeletal data to the same coordinate system) and remove noise (such as using filtering algorithms to process jitter data in motion capture) to ensure the comparability of real data and virtual data. By evaluating the action posture and facial expression and voice of the processed user behavior data and VR virtual character data, action similarity and facial expression and voice similarity are obtained. The weight coefficients W1 and W2 corresponding to action similarity and facial expression-voice similarity are retrieved. Both W1 and W2 are greater than zero. The behavior realism is calculated based on action similarity × W1 + facial expression-voice similarity × W2. It also judges whether the behavior realism reaches the preset behavior realism threshold. If it does, a qualified signal is generated; if it does not, an unqualified signal is generated. The visual feedback module is used to respond to qualified or unqualified signals, and immediately display the warning text "realistic behavior" corresponding to the qualified signal or the warning text "unrealistic behavior" corresponding to the unqualified signal, so as to intuitively understand the realism of the VR virtual character's behavior. The motion posture evaluation process is as follows: extract the skeletal key points (such as joint positions) of the VR virtual character and the user, obtain the point distance between the skeletal key points of the VR virtual character and the user, and set the point distance of the skeletal key points as the motion similarity; The facial expression and voice evaluation process is as follows: Obtain the user's facial expression feature image and the VR virtual character's virtual facial expression feature image. Set the similarity between the facial expression feature image and the virtual facial expression feature image as the expression similarity. Obtain the user's speech rate feature curve and pitch feature curve and the VR virtual character's speech rate feature curve and pitch feature curve. Set the similarity between the user's speech rate feature curve and the VR virtual character's speech rate feature curve and pitch feature curve as speech rate similarity and pitch similarity, respectively. Calculate the facial expression and voice similarity based on the weighted sum of expression similarity, speech rate similarity, and pitch similarity. That is, facial expression similarity × corresponding weight coefficient + speech rate similarity × corresponding weight coefficient + tone similarity × corresponding weight coefficient = facial expression and speech similarity; When generating a stable signal, the naturalness analysis module is used to perform behavioral naturalness acquisition and analysis on the collected VR virtual character's various actions. The specific behavioral naturalness acquisition and analysis process is as follows: SS1: Collect the time periods of each action of the VR virtual character (the duration between the start time and the end time of the action), divide the action time period into i time points, where i is a natural number greater than zero, and obtain the skeletal key point position data of the VR virtual character in each time period (such as the coordinates of the hip joint and elbow joint). SS2: Based on the position difference and time interval between two adjacent time points, the instantaneous velocity at each time point is obtained using the formula velocity = position difference / time interval, generating a time-instantaneous velocity change curve; SS3: Then, based on the velocity difference (instantaneous velocity difference) and time interval between two adjacent time points, use the formula acceleration = velocity difference / time interval to obtain the instantaneous acceleration at each time point and generate a time-acceleration change curve; SS4: Obtain the variance of the time-instantaneous velocity change curve and the time-acceleration change curve, obtain the maximum value of the variance of the time-instantaneous velocity change curve and the time-acceleration change curve, and set it as the smoothness index value; The smoothness index value is processed for discrimination. If the smoothness index value is less than the preset smoothness index value threshold, it is judged as non-smooth behavior. If the smoothness index value is greater than or equal to the preset smoothness index value threshold, it is judged as smooth behavior. The percentage of behaviors corresponding to behavior smoothing is obtained, and the percentage of behaviors corresponding to behavior smoothing is set as the behavior naturalness. The behavior naturalness is then judged. If the behavior naturalness is less than the preset behavior naturalness threshold, a non-compliance signal is generated. If the behavior naturalness is greater than or equal to the preset behavior naturalness threshold, a compliance signal is generated. The visual feedback module is used to respond to the qualified or unqualified signals, and immediately displays the warning text "natural behavior" corresponding to the qualified signal or the warning text "unnatural behavior" corresponding to the unqualified signal, so as to intuitively understand whether the behavior of the VR virtual character is smooth. The behavior generation improvement module is used to fuse, interact with, and summarize the output results of behavior realism acquisition analysis and behavior naturalness acquisition analysis. The specific fusion, interaction, and summary analysis process is as follows: Based on the interactive analysis of qualified signals, unqualified signals, qualified signals, and unqualified signals, output signals or updated signals are obtained; Interactive analysis: If a qualified signal and a compliant signal are generated, an output signal is obtained; if a qualified signal and a non-compliant signal, or a non-compliant signal and a compliant signal, or a non-compliant signal and a non-compliant signal are generated, an update signal is obtained. When an update signal is generated, the analysis results of the improved realism and naturalness of the VR virtual character's behavior are obtained until the output signal is generated. When generating the output signal, information such as the number of times the VR virtual character has been improved, the direction of improvement (e.g., facial feature improvement, voice improvement), and the corresponding values ​​of behavioral realism and naturalness for each number of improvements are obtained. A list of improvements for VR virtual character generation is constructed based on information such as the number of improvements, the direction of improvement (e.g., facial feature improvement, voice improvement, etc.), and the corresponding values ​​of behavioral realism and naturalness for each number of improvements. Send the list of improvements to VR virtual character generation to the virtual character generation center for storage; In order to ensure the rationality and naturalness of the generated behavior in the VR environment, it is also necessary to conduct multi-angle performance evaluation of the behavior. From the perspective of visual effects, it is necessary to evaluate whether the virtual character's movements are smooth and coordinated, and whether they conform to human movement habits and behavioral logic. It is also necessary to observe whether the arm swing and leg lifting movements of the virtual character when running are natural, and whether there are any stutters or incoordination. Through these performance evaluations, the behavior generation model and behavior mapping process can be continuously optimized to improve the quality of VR virtual character behavior and user experience.

[0019] Example 3: This invention also proposes a VR virtual character behavior generation method based on multimodal data fusion, including the following steps: Step 1: Multidimensional data collection and storage for users; Step 2: Data processing and behavior matching analysis of multidimensional data to obtain an input-behavior correspondence table; Step 3: Based on the progressive information behavior fit analysis process, compare and analyze the obtained objective matching ratios, and output stable signals or interference signals; Step 4: Analysis process for obtaining and analyzing the realism of behavior and the naturalness of behavior based on stable signals; Step 5: Based on the output results obtained from Step 4, perform fusion interaction and improvement summary analysis to obtain an improvement list for VR virtual character generation; In summary, different types of data complement each other to provide data support for a comprehensive understanding of user behavior. Standardized data processing and behavior binding processes help ensure the consistency between virtual character behavior and user intent, reducing interaction errors caused by behavior mismatch. Furthermore, real-time evaluation from an objective matching ratio perspective allows developers or users to promptly grasp the matching status between virtual character behavior and user input, avoiding the continuous impact of invalid interactions on user experience. It further ensures the rationality and naturalness of generated behavior in the VR environment. From a visual perspective, it evaluates whether the virtual character's movements are smooth, coordinated, and conform to human movement habits and behavioral logic. Through these performance evaluations, the behavior generation model is continuously optimized, ultimately achieving the goal of accurate interaction, high immersion, and a smooth experience.

[0020] The threshold is set for comparative analysis of results to determine whether they are good or bad. The value of the threshold is determined by a combination of large-scale model analysis of sample data and human experience. It can also be adjusted appropriately based on seasonal or common-sense influencing factors. The size of the coefficient is a specific value obtained by quantifying each parameter to facilitate subsequent comparison. The size of the coefficient depends on the amount of sample data and the corresponding operating coefficient initially set by those skilled in the art for each set of sample data; as long as it does not affect the proportional relationship between the parameter and the quantified value.

[0021] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A VR virtual character behavior generation system based on multimodal data fusion, characterized in that, It includes a virtual character generation center, a multimodal acquisition module, a behavior generation and matching module, a behavior fusion module, a behavior realism analysis module, a naturalness analysis module, a behavior generation improvement module, and a visual feedback module; The multimodal acquisition module collects multidimensional data from users through multiple sensors and sends it to the virtual character generation center for storage. The behavior generation and matching module is used to process and analyze the multidimensional data of retrieved users to obtain an input-behavior correspondence table. The behavior fusion module is used to analyze the behavior fit of user input data and character behavior data collected during the interaction between the user and the VR virtual character, and to perform discriminant analysis on the obtained objective matching ratio to obtain stable signals or interference signals. The stable signal-based behavior realism analysis module is used to analyze the behavior realism of collected user behavior data and VR virtual character data to obtain qualified or unqualified signals. The naturalness analysis module is used to analyze the naturalness of each action of the collected VR virtual character to obtain qualified or unqualified signals. The behavior generation improvement module is used to integrate, interact with, and summarize the output results of behavior realism acquisition analysis and behavior naturalness acquisition analysis to obtain a list of improvements for VR virtual character generation.

2. The VR virtual character behavior generation system based on multimodal data fusion according to claim 1, characterized in that, The analysis process of the behavior generation and matching module is as follows: S1: Preprocess and fuse the collected multidimensional data from users to obtain multimodal fusion data. Preprocessing includes cleaning and standardization. S2: Generates behavioral decisions for virtual characters based on multimodal fusion data and a pre-set behavior generation model. Behavioral decisions include action commands and facial expression commands. S3: Organize user input methods and specific commands into a list; S4: Based on the list, bind a unique correct role behavior to each input instruction to form an input-behavior mapping table.

3. The VR virtual character behavior generation system based on multimodal data fusion according to claim 1, characterized in that, The analysis process of the behavior fusion module is as follows: Acquire user input data and character behavior data during the interaction between the user and the VR virtual character; User input data and character behavior data within the same interaction cycle are paired using timestamps. An interaction cycle is defined as the period from the start of user input to the end of VR virtual character behavior. n sets of user-VR virtual character interaction cycles are obtained, where n is a natural number greater than zero. The user input data and character behavior data are checked and paired one by one according to the input-behavior mapping table to obtain the pairing results. The pairing results include correct matches and incorrect matches. If the behavior of the VR virtual character matches the input-behavior correspondence table, it is judged as a correct match; if the behavior of the VR virtual character does not match the input-behavior correspondence table, it is judged as an incorrect match.

4. The VR virtual character behavior generation system based on multimodal data fusion according to claim 3, characterized in that, Based on the pairing results, the number of correct matches is set as N1, and the ratio between N1 and the total number of interactions is set as the objective matching ratio. The objective matching ratio is then judged to see if it exceeds the preset objective matching ratio threshold. If it does, a stable signal is generated; if it does not, an interference signal is generated.

5. The VR virtual character behavior generation system based on multimodal data fusion according to claim 1, characterized in that, The analysis process of the behavior realism analysis module is as follows: Collect user behavior data and VR virtual character data, and perform data format unification and noise removal on the collected user behavior data and VR virtual character data; By evaluating the action posture and facial expression and voice of the processed user behavior data and VR virtual character data, action similarity and facial expression and voice similarity are obtained. The weight coefficients W1 and W2 corresponding to action similarity and facial expression-voice similarity are retrieved. Both W1 and W2 are greater than zero. The behavior realism is calculated based on action similarity × W1 + facial expression-voice similarity × W2. The system determines whether the behavior realism reaches a preset behavior realism threshold, and obtains a qualified signal or an unqualified signal.

6. The VR virtual character behavior generation system based on multimodal data fusion according to claim 1, characterized in that, The analysis process of the naturalness analysis module is as follows: SS1: Collect the time periods of each action of the VR virtual character (the duration between the start time and the end time of the action), divide the action time period into i time points, where i is a natural number greater than zero, and obtain the skeletal key point position data of the VR virtual character in each time period. SS2: Based on the position difference and time interval between two adjacent time points, the instantaneous velocity at each time point is obtained using the formula velocity = position difference / time interval, generating a time-instantaneous velocity change curve; SS3: Then, based on the velocity difference and time interval between two adjacent time points, use the formula acceleration = velocity difference / time interval to obtain the instantaneous acceleration at each time point and generate a time-acceleration change curve; SS4: Obtain the maximum value of the variance between the time-instantaneous velocity change curve and the time-acceleration change curve, and set it as the smoothness index value; The smoothness index value is processed to determine whether the behavior is non-smooth or smooth. Obtain the percentage of behaviors corresponding to behavior smoothing, set the percentage of behaviors corresponding to behavior smoothing as the behavior naturalness, and perform discrimination processing on the behavior naturalness to obtain the non-compliant signal or compliant signal.

7. The VR virtual character behavior generation system based on multimodal data fusion according to claim 1, characterized in that, The analysis process of the behavior generation improvement module is as follows: Based on the interactive analysis of qualified signals, unqualified signals, qualified signals, and unqualified signals, output signals or updated signals are obtained; When an update signal is generated, the analysis results of the improved realism and naturalness of the VR virtual character's behavior are obtained until the output signal is generated. When the output signal is generated, information such as the number of times the VR virtual character has been improved and the direction of improvement is obtained; An improvement list for VR virtual characters is generated based on the number of improvements and the direction of improvement.

8. A VR virtual character behavior generation method based on multimodal data fusion, wherein the method is applied to a VR virtual character behavior generation system based on multimodal data fusion as described in any one of claims 1-7, characterized in that, Includes the following steps: Step 1: Multidimensional data collection and storage for users; Step 2: Data processing and behavior matching analysis of multidimensional data to obtain an input-behavior correspondence table; Step 3: Based on the progressive information behavior fit analysis process, compare and analyze the obtained objective matching ratios, and output stable signals or interference signals; Step 4: Analysis process for obtaining and analyzing the realism of behavior and the naturalness of behavior based on stable signals; Step 5: Based on the output results obtained from Step 4, perform fusion interaction and improvement summary analysis to obtain an improvement list for VR virtual character generation.

Images