Virtual character interaction design generation method based on multimodal perception

Abstract

The present invention belongs to the field of virtual character generation and relates to interactive testing technology, which is used to solve the problem that the existing technology cannot generate virtual characters for simulation training. Specifically, it is a virtual character interaction design generation method based on multimodal perception, including a character generation sub-method and a test optimization sub-method; the character generation sub-method includes the following steps: "Virtual image setting conception": setting basic parameters of the virtual character, the basic parameters include image style, category, positioning and basic information, the basic information includes gender, age, appearance characteristics and personality characteristics, realizing first-person perspective roaming and device interactive operation, allowing users to operate virtual devices; the present invention can simulate gas stations, oil depots and other depot scenes, and can also develop emergency accident scenes for simulation training. At the same time, it combines key technologies such as 3D modeling and emergency simulation to ensure training effects.

Description

Technical Field

[0001] The present invention belongs to the field of virtual character generation, and relates to an interactive testing technology, in particular to a virtual character interactive design generation method based on multimodal perception. Background Art

[0002] With the development of the petrochemical industry, the operating procedures of grassroots depots and stations are complicated and the safety risks are high. Traditional training methods are difficult to meet the requirements of modern technology and safe production, especially on-site training, which brings great costs and risks to enterprises due to complex equipment and environmental safety issues.

[0003] The invention patent with publication number CN112950419A discloses a wind farm wind turbine operation and maintenance virtual training and emergency drill system. The emergency drill system can improve the technical level of operation and maintenance personnel, has practical promotion and application value, can provide information-based auxiliary means and tools for wind power operation management, and can further promote the improvement of wind farm operation management technology level and management efficiency; however, the emergency drill system cannot generate virtual characters for simulation training, nor can it perform interactive testing and analysis on the generated virtual characters, resulting in low immersion and interaction levels in the simulation training, and the training effect cannot be guaranteed.

[0004] In view of the above technical problems, this application proposes a solution. Summary of the invention

[0005] The purpose of the present invention is to provide a virtual character interaction design generation method based on multimodal perception, which is used to solve the problem that the prior art cannot generate virtual characters for simulation training.

[0006] The technical problem to be solved by the present invention is: how to provide a virtual character interaction design generation method based on multimodal perception that can generate virtual characters for simulation training.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] A virtual character interaction design generation method based on multimodal perception, including a character generation sub-method and a test optimization sub-method;

[0009] The character generation sub-method includes the following steps:

[0010] Step S1: ‌Avatar setting concept‌: setting basic parameters of the avatar, including image style, category, positioning and basic information, including gender, age, appearance and personality characteristics;

[0011] Step S2: Perform 2D image design: design a 2D image according to the basic parameters of the virtual character;

[0012] Step S3: Perform 3D modeling design;

[0013] Step S4: ‌3D model binding‌: After completing 3D modeling, select key points, map the identified key points to the model for binding, and import the model into Unity3D;

[0014] Step S5: Interactive function development: realizing first-person perspective roaming and device interactive operation, allowing users to operate virtual devices;

[0015] Step S6: Content production and driving: The whole body motion capture is completed through the inertial motion capture equipment, optical motion capture system and facial capture system. At the same time, it is driven by AI algorithm to enable the virtual character to interact with the user.

[0016] Furthermore, in step S3, 3D modeling is performed using 3ds Max, and a virtual scene is constructed using Unity3D to model the human body structure, facial features, and clothing, and to add materials and textures.

[0017] Further, the test optimization sub-method includes the following steps:

[0018] Step P1: Performing interactive testing on the virtual character: marking the generated virtual character as a test object, generating a test cycle, executing several interactive test processes within the test cycle, and randomly selecting a test data group from the instruction library at the beginning of the interactive test process. The test data group includes a test instruction, a standard action combination corresponding to the test instruction, and a standard execution time. The types of test instructions include voice instructions and gesture instructions, and the standard action combination includes several standard sub-actions;

[0019] Step P2: Perform data statistics on the interactive test process and obtain the execution response data ZX, execution deviation data ZP and execution error data ZW of the interactive test process;

[0020] Step P3: Perform data processing and analysis on the interactive test process: perform numerical calculations on the execution response data ZX, execution deviation data ZP, and execution error data ZW of the interactive test process to obtain the test coefficient CS; sum and average the test coefficients CS of all interactive test processes in the test cycle to obtain the test performance value; perform variance calculation on the test coefficients of all interactive test processes in the test cycle to obtain the test concentration value;

[0021] Step P4: Evaluate the interactive test results and determine whether the interactive test results of the test object meet the requirements. If they do not meet the requirements, execute step P5;

[0022] Step P5: Optimize and analyze the interactive performance and determine whether the interactive test of the test object has a tendency. If there is a tendency, execute step P6;

[0023] Step P6: Optimize decision-making processing on the test object.

[0024] Furthermore, the process of acquiring the execution response data ZX, the execution deviation data ZP and the execution error data ZW includes: demonstrating the test instruction to the test object, marking the end time of the demonstration as the receiving time, then marking the time when the test object starts to execute the action as the execution time, and marking the time difference between the execution time and the receiving time as the execution response data ZX of the interactive test process; marking the time when the test object completes the action execution as the completion time, marking the time difference between the completion time and the execution time as the execution value, and marking the absolute value of the difference between the execution value and the standard execution time as the execution deviation data ZP of the interactive test process; dividing the execution action combination of the test object into several execution sub-actions, and comparing the execution sub-action with all standard sub-actions in the standard action combination: if the standard action combination contains a standard sub-action that is the same as the execution sub-action, then the corresponding execution sub-action is marked as a compliant sub-action; if the standard action combination does not contain a standard sub-action that is the same as the execution sub-action, then the corresponding execution sub-action is marked as an error sub-action; and the ratio of the number of error sub-actions to the number of execution sub-actions is marked as the execution error data ZW of the interactive test process.

[0025] Furthermore, the specific process of evaluating the interactive test results includes: obtaining the test performance threshold and test concentration threshold of the test object, and comparing the test performance value and test concentration value of the test object with the test performance threshold and test concentration threshold respectively: if the test performance value is less than the test performance threshold and the test concentration value is less than the test concentration threshold, then it is determined that the interactive test result of the test object meets the requirements, and an interactive qualified signal is generated and sent to the administrator's mobile phone terminal; otherwise, it is determined that the interactive test result of the test object does not meet the requirements.

[0026] Furthermore, the specific process of optimizing and analyzing the interactive performance includes: marking the interactive test process whose test coefficient CS is not less than the test performance threshold as an abnormal process, marking the abnormal process using voice instructions as a voice tendency process, marking the abnormal process using gesture instructions as a gesture tendency process, marking the absolute value of the difference between the number of voice tendency processes and the number of gesture tendency processes as a tendency performance value, and judging whether the interactive test of the test object has a tendency through the tendency performance value.

[0027] Furthermore, the specific process of determining whether the interactive test of the test object has a tendency includes: comparing the tendency expression value with a preset tendency expression threshold: if the tendency expression value is less than the tendency expression threshold, then the interactive test of the test object is determined to have no instruction tendency, generating an identification optimization signal and sending the identification optimization signal to the mobile phone terminal of the administrator; if the tendency expression value is greater than or equal to the tendency expression threshold, then the interactive test of the test object is determined to have an instruction tendency.

[0028] Furthermore, the specific process of optimizing the decision-making process for the test object includes: comparing the number of speech tendency processes with the number of gesture tendency processes: if the number of speech tendency processes is greater than the number of gesture tendency processes, generating a speech optimization signal and sending the speech optimization signal to the mobile phone terminal of the administrator; if the number of gesture tendency processes is greater than the number of speech tendency processes, generating a gesture optimization signal and sending the gesture optimization signal to the mobile phone terminal of the administrator.

[0029] The present invention has the following beneficial effects:

[0030] The role generation sub-method can be used to simulate gas stations, oil depots and other depot scenes, and can also develop emergency accident scenes for simulation training. At the same time, key technologies such as 3D modeling and emergency simulation can be combined to ensure the training effect;

[0031] Conduct interactive testing on virtual characters. During the interactive testing process, the test data of the test object are statistically analyzed and comprehensively processed to obtain the test coefficient. Then, the interactive test results of the test object are evaluated in combination with the test coefficients of all interactive testing processes to ensure the interactive performance of the virtual character.

[0032] The interaction performance is optimized and analyzed. After the abnormal processes are marked, the frequency of voice and gesture commands of the abnormal processes is analyzed to obtain the tendency performance value. The tendency performance value, the number of voice tendency processes, and the number of gesture tendency processes are combined to generate targeted optimization signals to improve the interaction optimization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 This is a flow chart of a character generation sub-method according to the first embodiment of the present invention;

[0035] Figure 2This is a flowchart of the test optimization sub-method of the second embodiment of the present invention. DETAILED DESCRIPTION

[0036] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] A virtual character interaction design generation method based on multimodal perception, including a character generation sub-method and a test optimization sub-method;

[0038] Embodiment 1: Figure 1 As shown, the character generation sub-method includes the following steps:

[0039] Step S1: ‌Avatar setting concept‌: setting basic parameters of the avatar, including image style, category, positioning and basic information, including gender, age, appearance and personality characteristics;

[0040] Step S2: Perform 2D image design: design a 2D image according to the basic parameters of the virtual character;

[0041] Step S3: 3D modeling and design: 3D modeling is performed using 3ds Max, and a virtual scene is built using Unity3D. The human body structure, facial features, clothing, etc. are modeled, and materials and textures are added to increase the realism of the virtual character;

[0042] Step S4: ‌3D model binding‌: After completing 3D modeling, you need to bind the skeleton to the model so that it can perform various actions and expressions. This includes selecting key points and mapping the identified key points to the model for binding, importing the model into Unity3D, creating environmental elements, and enhancing the realism of the scene. In Unity3D, the creation of virtual characters usually begins with the use of professional 3D modeling tools. After importing the model into Unity3D, you need to configure the material and scene, which includes setting the appropriate material, texture, light and shadow effects for the model to ensure its visual effect in the game. At the same time, you also need to configure the scene, including setting the camera, lighting, terrain, etc., to build a realistic virtual environment‌;

[0043] Step S5: Interactive function development: realizing first-person perspective roaming and device interactive operation, allowing users to operate virtual devices, such as starting, stopping, and adjusting parameters;

[0044] Step S6: Content production and driving: Through the whole-body inertial motion capture equipment, optical motion capture system and facial capture system, the whole-body motion capture is completed to realize the real-time changes of body, expression and gesture. At the same time, it can be driven by AI algorithm to enable the virtual character to understand language and environment and interact with users; the character generation sub-method can be used to simulate gas stations, oil depots and other depot scenes, and emergency accident scenes can also be developed for simulation training. At the same time, key technologies such as 3D modeling and emergency simulation are combined to ensure the training effect.

[0045] Embodiment 2: Figure 2 As shown, the test optimization sub-method includes the following steps:

[0046] Step P1: Performing interactive testing on the virtual character: marking the generated virtual character as a test object, generating a test cycle, executing several interactive test processes within the test cycle, and randomly selecting a test data group from the instruction library at the beginning of the interactive test process. The test data group includes a test instruction, a standard action combination corresponding to the test instruction, and a standard execution time. The types of test instructions include voice instructions and gesture instructions, and the standard action combination includes several standard sub-actions;

[0047] Step P2: Perform data statistics on the interactive test process: perform a test instruction demonstration on the test object, mark the end time of the demonstration as the receiving time, then mark the time when the test object starts to execute the action as the execution time, and mark the time difference between the execution time and the receiving time as the execution response data ZX of the interactive test process; mark the time when the test object completes the action execution as the completion time, mark the time difference between the completion time and the execution time as the execution value, and mark the absolute value of the difference between the execution value and the execution standard duration as the execution deviation data ZP of the interactive test process; divide the execution action combination of the test object into several execution sub-actions, and compare the execution sub-action with all standard sub-actions in the standard action combination: if the standard action combination contains a standard sub-action that is the same as the execution sub-action, then mark the corresponding execution sub-action as a compliance sub-action; if the standard action combination does not contain a standard sub-action that is the same as the execution sub-action, then mark the corresponding execution sub-action as an error sub-action; mark the ratio of the number of error sub-actions to the number of execution sub-actions as the execution error data ZW of the interactive test process;

[0048] Step P3: Perform data processing and analysis on the interactive test process: The test coefficient CS of the interactive test process is obtained by the formula CS=k1×ZX+k2×ZP+k3×ZW, where k1, k2 and k3 are all proportional coefficients, and k3>k2>k1>1; the test coefficients CS of all interactive test processes in the test cycle are summed and averaged to obtain the test performance value, and the test coefficients of all interactive test processes in the test cycle are calculated to obtain the test concentration value; in the interactive test process, various test data of the test object are statistically and comprehensively processed to obtain the test coefficient, and then the interactive test results of the test object are evaluated in combination with the test coefficients of all interactive test processes to ensure the interactive performance of the virtual character;

[0049] Step P4: Evaluate the interactive test result: obtain the test performance threshold and test concentration threshold of the test object, and compare the test performance value and test concentration value of the test object with the test performance threshold and test concentration threshold respectively: if the test performance value is less than the test performance threshold and the test concentration value is less than the test concentration threshold, it is determined that the interactive test result of the test object meets the requirements, and an interactive qualified signal is generated and sent to the mobile phone terminal of the administrator; otherwise, it is determined that the interactive test result of the test object does not meet the requirements, and step P5 is executed;

[0050] Step P5: Optimize and analyze the interactive performance: mark the interactive test process whose test coefficient CS is not less than the test performance threshold as an abnormal process, mark the abnormal process using voice instructions as a voice tendency process, mark the abnormal process using gesture instructions as a gesture tendency process, mark the absolute value of the difference between the number of voice tendency processes and the number of gesture tendency processes as a tendency performance value, and compare the tendency performance value with the preset tendency performance threshold: if the tendency performance value is less than the tendency performance threshold, it is determined that the interactive test of the test object does not have a command tendency, generate an identification optimization signal and send the identification optimization signal to the mobile phone terminal of the administrator; if the tendency performance value is greater than or equal to the tendency performance threshold, it is determined that the interactive test of the test object has a command tendency, and execute step P6;

[0051] Step P6: Optimize the decision-making process for the test object: compare the number of voice tendency processes with the number of gesture tendency processes: if the number of voice tendency processes is greater than the number of gesture tendency processes, generate a voice optimization signal and send the voice optimization signal to the administrator's mobile phone terminal; if the number of gesture tendency processes is greater than the number of voice tendency processes, generate a gesture optimization signal and send the gesture optimization signal to the administrator's mobile phone terminal; after marking the abnormal process, analyze the frequency of voice instructions and gesture instructions of the abnormal process to obtain the tendency expression value, and generate targeted optimization signals based on the tendency expression value, the number of voice tendency processes and the number of gesture tendency processes to improve the efficiency of interaction optimization.

[0052] The method for interactive design and generation of virtual characters based on multimodal perception sets the basic parameters of the virtual characters, designs a 2D image according to the basic parameters of the virtual characters, uses 3ds Max for 3D modeling, uses Unity3D to build virtual scenes, models the human body structure, facial features, clothing, etc., realizes first-person perspective roaming and device interactive operations, and allows users to operate virtual devices, such as starting, stopping, and adjusting parameters; conducts interactive tests on virtual characters, performs data processing and analysis on the interactive test process, and obtains test performance values ​​and test concentration values; evaluates the interactive test results and performs optimization analysis when the interactive test results do not meet the requirements.

[0053] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

[0054] The above formulas are obtained by collecting a large amount of data for software simulation and selecting a formula close to the real value. The coefficients in the formula are set by technicians in this field according to the actual situation; for example: formula CS=k1×ZX+k2×ZP+k3×ZW; technicians in this field collect multiple groups of sample data and set corresponding test coefficients for each group of sample data; substitute the set test coefficients and the collected sample data into the formula, any three formulas constitute a three-variable linear equation group, screen the calculated coefficients and take the average, and obtain the values ​​of k1, k2 and k3 as 4.39, 2.85 and 2.23 respectively;

[0055] The size of the coefficient is to quantify each parameter to obtain a specific value for subsequent comparison. The size of the coefficient depends on the amount of sample data and the preliminary setting of the corresponding test coefficient for each set of sample data by technical personnel in this field; as long as it does not affect the proportional relationship between the parameter and the quantized value, such as the test coefficient is proportional to the value of the execution response data.

[0056] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0057] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A virtual character interaction design generation method based on multimodal perception, characterized in that: Includes role generation sub-method and test optimization sub-method; The character generation sub-method includes the following steps: Step S1: ‌Avatar setting concept‌: setting basic parameters of the avatar, including image style, category, positioning and basic information, including gender, age, appearance and personality characteristics; Step S2: Designing a 2D image: Designing a 2D image based on the basic parameters of the virtual character; Step S3: 3D modeling and design: 3D modeling is performed using 3ds Max, and a virtual scene is built using Unity3D to model the human body structure, facial features, and clothing, and to add materials and textures; Step S4: ‌3D model binding‌: After completing 3D modeling, select key points, map the identified key points to the model for binding, and import the model into Unity3D; Step S5: Interactive function development: realizing first-person perspective roaming and device interactive operation, allowing users to operate virtual devices; Step S6: Content production and driving: The whole body motion capture is completed through the inertial motion capture device, optical motion capture system and facial capture system. At the same time, it is driven by AI algorithm to enable the virtual character to interact with the user; The test optimization sub-method includes the following steps: Step P1: Performing interactive testing on the virtual character: marking the generated virtual character as a test object, generating a test cycle, executing several interactive test processes within the test cycle, and randomly selecting a test data group from the instruction library at the beginning of the interactive test process. The test data group includes a test instruction, a standard action combination corresponding to the test instruction, and a standard execution time. The types of test instructions include voice instructions and gesture instructions, and the standard action combination includes several standard sub-actions; Step P2: Perform data statistics on the interactive test process and obtain the execution response data ZX, execution deviation data ZP and execution error data ZW of the interactive test process; Step P3: Perform data processing and analysis on the interactive test process: perform numerical calculations on the execution response data ZX, execution deviation data ZP and execution error data ZW of the interactive test process, and obtain the test coefficient CS of the interactive test process through the formula CS=k1×ZX+k2×ZP+k3×ZW, where k1, k2 and k3 are all proportional coefficients, and k3>k2>k1>1; sum and average the test coefficients CS of all interactive test processes in the test cycle to obtain the test performance value, and perform variance calculation on the test coefficients of all interactive test processes in the test cycle to obtain the test concentration value; Step P4: Evaluate the interactive test results and determine whether the interactive test results of the test object meet the requirements. If they do not meet the requirements, execute step P5; Step P5: Optimize and analyze the interactive performance and determine whether the interactive test of the test object has a tendency. If there is a tendency, execute step P6; Step P6: Optimize decision-making processing on the test object.

2. The method for generating virtual character interaction design based on multimodal perception according to claim 1, characterized in that: The acquisition process of execution response data ZX, execution deviation data ZP and execution error data ZW includes: demonstrating the test instruction to the test object, marking the end time of the demonstration as the receiving time, then marking the time when the test object starts to execute the action as the execution time, and marking the time difference between the execution time and the receiving time as the execution response data ZX of the interactive test process; marking the time when the test object completes the action execution as the completion time, marking the time difference between the completion time and the execution time as the execution value, and marking the absolute value of the difference between the execution value and the execution standard time length as the execution deviation data ZP of the interactive test process; dividing the execution action combination of the test object into several execution sub-actions, and comparing the execution sub-action with all standard sub-actions in the standard action combination: if the standard action combination contains a standard sub-action that is the same as the execution sub-action, then the corresponding execution sub-action is marked as a compliant sub-action; if the standard action combination does not contain a standard sub-action that is the same as the execution sub-action, then the corresponding execution sub-action is marked as an error sub-action; and the ratio of the number of error sub-actions to the number of execution sub-actions is marked as the execution error data ZW of the interactive test process.

3. The method for generating virtual character interaction design based on multimodal perception according to claim 2, characterized in that: The specific process of evaluating the interactive test results includes: obtaining the test performance threshold and test concentration threshold of the test object, and comparing the test performance value and test concentration value of the test object with the test performance threshold and test concentration threshold respectively: if the test performance value is less than the test performance threshold and the test concentration value is less than the test concentration threshold, then it is determined that the interactive test result of the test object meets the requirements, and an interactive qualified signal is generated and sent to the administrator's mobile phone terminal; otherwise, it is determined that the interactive test result of the test object does not meet the requirements.

4. The method for generating virtual character interaction design based on multimodal perception according to claim 3 is characterized in that: The specific process of optimizing and analyzing the interactive performance includes: marking the interactive test process whose test coefficient CS is not less than the test performance threshold as an abnormal process, marking the abnormal process using voice instructions as a voice-oriented process, marking the abnormal process using gesture instructions as a gesture-oriented process, marking the absolute value of the difference between the number of voice-oriented processes and the number of gesture-oriented processes as a tendency performance value, and judging whether the interactive test of the test object has a tendency through the tendency performance value.

5. The method for generating virtual character interaction design based on multimodal perception according to claim 4, characterized in that: The specific process of determining whether the interactive test of the test object has a tendency includes: comparing the tendency expression value with the preset tendency expression threshold: if the tendency expression value is less than the tendency expression threshold, then the interactive test of the test object is determined to have no instruction tendency, generating an identification optimization signal and sending the identification optimization signal to the mobile phone terminal of the administrator; if the tendency expression value is greater than or equal to the tendency expression threshold, then the interactive test of the test object is determined to have an instruction tendency.

6. The method for generating virtual character interaction design based on multimodal perception according to claim 5, characterized in that: The specific process of optimizing the decision-making process for the test object includes: comparing the number of speech tendency processes with the number of gesture tendency processes: if the number of speech tendency processes is greater than the number of gesture tendency processes, generating a speech optimization signal and sending the speech optimization signal to the manager's mobile phone terminal; if the number of gesture tendency processes is greater than the number of speech tendency processes, generating a gesture optimization signal and sending the gesture optimization signal to the manager's mobile phone terminal.

Images