Experimental simulation interaction method and system based on virtual reality technology

Abstract

The present application provides an experimental simulation interaction method and system based on virtual reality technology, which relates to the field of computer data processing technology. The method includes: parsing multiple target experimental element information in a target experimental course; for each target experimental element information, determining a matching target experimental element component from a virtual experimental environment according to the target experimental element information, inputting the target experimental element component into a confusion component generation model to obtain a confusion experimental element component corresponding to the target experimental element component; setting the confusion experimental element component in the virtual experimental environment according to the first spatial coordinates of each adjacent position; detecting user interaction operations according to each target experimental element component and each confusion experimental element component to evaluate the user's recognition ability of each target experimental element information. Thus, the effect of experimental teaching is enhanced based on similar confusion experimental element components, which helps to evaluate students' mastery of experimental key points.

Description

Technical Field

[0001] The present application relates to the field of computer data processing technology, and in particular to an experimental simulation interaction method and system based on virtual reality technology. Background Art

[0002] With the rapid development of virtual reality (VR) technology, education and training methods based on VR technology have gradually received widespread attention and application. VR technology provides users with an immersive interactive experience by creating a highly simulated virtual environment, allowing complex experimental processes and phenomena to be presented to users in an intuitive way. In scientific experimental teaching, the virtual experimental environment can effectively simulate the actual experimental scenes and provide a safe, efficient and repeatable experimental experience.

[0003] Traditional experimental teaching usually requires a large amount of experimental equipment, chemical reagents and other resources, and there may be certain safety risks during the experiment. In addition, due to limited experimental resources, students' experimental operation opportunities and hands-on opportunities are limited, and they cannot fully master experimental skills and understand experimental principles. In order to solve these problems, virtual experimental systems came into being, reproducing the experimental process in a digital and virtual way, thereby reducing resource consumption and improving experimental teaching efficiency.

[0004] However, current virtual experiment systems are usually implemented through a series of preset fixed scenes and interactive methods. Although this method can simulate the operation process of real experiments, its flexibility and interactivity are often insufficient, and it is difficult to fully meet the diverse needs of different experimental courses. In addition, the existing virtual experiment systems usually lack effective interference and confusion mechanisms in the configuration and interaction of experimental elements. When students conduct virtual experiments, it is difficult to deeply understand the key points of the experimental content, resulting in deficiencies in the personalized interactivity of experiments and the cultivation of enhanced capabilities.

[0005] In response to the above problems, the industry has not yet proposed a better technical solution. Summary of the invention

[0006] The present application provides an experimental simulation interaction method, system, storage medium, computer program product and electronic device based on virtual reality technology, which are used to at least solve the problems of limited interaction mode and weak experimental teaching and training effect caused by preset fixed virtual experimental scenes.

[0007] In a first aspect, an embodiment of the present application provides an experimental simulation interaction method based on virtual reality technology, comprising: parsing multiple target experimental element information in a target experimental course; the experimental elements include any one of the following: experimental instruments, reactants and catalysts; for each of the target experimental element information, determining a matching target experimental element component from a virtual experimental environment according to the target experimental element information, and inputting the target experimental element component into a confusion component generation model to obtain a confusion experimental element component corresponding to the target experimental element component; the experimental element information corresponding to the confusion experimental element component is different from the target experimental element information; multiple experimental element components are deployed in the virtual experimental environment, and each of the experimental element components is deployed at a corresponding spatial position; the confusion component generation model adopts a generative adversarial network; respectively determine each first spatial coordinate adjacent to the spatial position of each of the target experimental element components, and set the confusion experimental element component in the virtual experimental environment according to each of the first spatial coordinates; detect user interaction operations according to each of the target experimental element components and each of the confusion experimental element components in the virtual experimental environment to evaluate the user's recognition ability of each of the target experimental element information in the target experimental course.

[0008] In a second aspect, an embodiment of the present application provides an experimental simulation interaction system based on virtual reality technology, including: a target element parsing unit, used to parse multiple target experimental element information in a target experimental course; the experimental elements include any one of the following: experimental instruments, reactants and catalysts; a confusion element generation unit, used to determine a matching target experimental element component from a virtual experimental environment according to the target experimental element information for each of the target experimental element information, and input the target experimental element component into a confusion component generation model to obtain a confusion experimental element component corresponding to the target experimental element component; the experimental element information corresponding to the confusion experimental element component is different from the target experimental element information information; multiple experimental element components are deployed in the virtual experimental environment, and each of the experimental element components is deployed in a corresponding spatial position; the confusion component generation model adopts a generative adversarial network; a confusion element setting unit is used to respectively determine each first spatial coordinate adjacent to the spatial position of each target experimental element component, and set the confusion experimental element component in the virtual experimental environment according to each first spatial coordinate; an element interaction detection unit is used to detect user interaction operations according to each target experimental element component and each confusion experimental element component in the virtual experimental environment, so as to evaluate the user's ability to recognize each target experimental element information in the target experimental course.

[0009] In a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the experimental simulation interaction method based on virtual reality technology of any embodiment of the present application.

[0010] In a fourth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, the steps of the experimental simulation interaction method based on virtual reality technology of any embodiment of the present application are implemented.

[0011] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the experimental simulation interaction method based on virtual reality technology of any embodiment of the present application.

[0012] The experimental simulation interaction method based on virtual reality technology provided by this application can produce at least the following technical effects:

[0013] (1) By parsing the information of multiple target experimental elements in the target experimental course, it is possible to deploy multiple experimental element components in the virtual experimental environment according to the needs of different experimental courses, and set confusing experimental element components in the corresponding adjacent spatial positions. Compared with the traditional method of pre-setting fixed scenes, it can adaptively meet the diverse needs of different experimental courses and provide richer and more interactive experimental scenes, so that students can have a more interactive learning experience when performing experimental operations, improve students' practical operation ability, and make their understanding of experimental knowledge not limited to the impression of textbook text content.

[0014] (2) Through the obfuscated component generation model, the generative adversarial network is used to generate obfuscated experimental element components corresponding to the target experimental element components. By deploying these obfuscated experimental element components in a spatial position adjacent to the target experimental element components, students need to identify different experimental elements when conducting virtual experiments, thereby deepening their understanding of the experimental content. Subsequently, by detecting user interaction operations and evaluating the user's ability to identify the information of each target experimental element in the target experimental course, it can help students improve their experimental skills and understanding of experimental principles in simulated experimental operations.

[0015] (3) By introducing confusing experimental elements into the virtual experimental environment, the complexity and challenge of the experimental operation are increased, forcing students to carefully identify and judge during the experiment. This can effectively enhance students' ability to identify experimental element information and improve their depth and comprehensiveness of understanding of the experimental content, thereby better mastering experimental skills and understanding experimental principles.

[0016] (4) Since the experimental elements in the virtual experimental environment can be reused, students can perform experimental operations multiple times at different times and places and fully master experimental skills. In addition, the obfuscated experimental elements generated by the generative adversarial network make each experimental process have a certain degree of randomness and uniqueness, which can provide personalized experimental experience for different courses.

[0017] Through this technical solution, by parsing and configuring the target experimental element information and deploying the corresponding experimental element components and obfuscated experimental element components in the virtual experimental environment, a virtual experimental scene can be quickly built. Students can practice and operate repeatedly in the virtual experimental environment. The obfuscated experimental element components enhance the effect of experimental teaching, help students master experimental skills and knowledge points in a short time, and improve the overall efficiency of experimental teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flowchart showing an example of an experimental simulation interaction method based on virtual reality technology according to an embodiment of the present application is shown;

[0020] Figure 2 A flowchart showing another example of an experimental simulation interaction method based on virtual reality technology according to an embodiment of the present application is shown;

[0021] Figure 3 A schematic diagram of structural connection of an example of an obfuscation component generation model according to an embodiment of the present application is shown;

[0022] Figure 4 A processing flow chart of an example of a determination operation for obfuscated description information according to an embodiment of the present application is shown;

[0023] Figure 5 A structural block diagram of an example of an experimental simulation interaction system based on virtual reality technology according to an embodiment of the present application is shown;

[0024] Figure 6 It is a schematic structural diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0027] Figure 1 A flowchart of an example of an experimental simulation interaction method based on virtual reality technology according to an embodiment of the present application is shown.

[0028] Regarding the executor of the method of the embodiment of the present application, it can be any controller or processor with computing or processing capabilities. By parsing and configuring the target experimental element information, flexibly setting the experimental element components and confusing experimental element components in the virtual experimental environment, detecting and evaluating the user's interactive operations, the interactivity and flexibility of the virtual experiment are enhanced, the students' understanding and identification of the experimental content are improved, the consumption of experimental resources and safety risks are reduced, and the efficiency of experimental teaching is improved.

[0029] In some examples, it can be a VR experiment simulation platform, and can be integrated and configured in an electronic device or terminal through software, hardware, or a combination of software and hardware, and the type of terminal or electronic device can be diverse, such as a mobile phone, tablet computer, or desktop computer, etc.

[0030] like Figure 1 As shown, in step S110, multiple target experiment element information in the target experiment course is parsed, and the experiment element includes any one of the following: experimental instruments, reactants, and catalysts.

[0031] In some embodiments, after a teacher or student inputs or selects a target experimental course into the system, the content of the experimental course can be semantically parsed through natural language processing technology and knowledge graphs to identify and extract detailed information on the experimental elements, which may include descriptive information of experimental elements such as experimental instruments (such as beakers, test tubes), reactants (such as specific chemical reagents), and catalysts, such as model, size, color, etc., and the extracted experimental element information can be classified and organized to form structured data.

[0032] In step S120, for each target experiment element information, a matching target experiment element component is determined from the virtual experiment environment according to the target experiment element information, and the target experiment element component is input into the obfuscation component generation model to obtain an obfuscation experiment element component corresponding to the target experiment element component.

[0033] Here, the experimental element information corresponding to the obfuscated experimental element component is different from the target experimental element information. A plurality of experimental element components are deployed in the virtual experimental environment, and each experimental element component is deployed in a corresponding spatial position.

[0034] In some embodiments, the information of the obfuscated experimental element components includes, but is not limited to, information such as shape, color, physical state, etc. that is different from the target experimental elements. In addition, according to the target experimental element information parsed from the target experimental course, the corresponding experimental element components are matched from the virtual experimental environment, which can be the corresponding experimental instrument models, reactants and catalysts in the virtual experimental scene. The retrieved target experimental element components are input into the obfuscated component generation model. The obfuscated component generation model uses a generative adversarial network (GAN) to generate obfuscated experimental element components that are similar to but different from the target experimental element components through adversarial training of the generator and the discriminator, such as a slanted bottom beaker and a round bottom beaker, a blue solution and a purple solution, and so on.

[0035] In step S130, first spatial coordinates adjacent to the spatial positions of the target experimental element components are determined respectively, and the obfuscated experimental element components are set in the virtual experimental environment according to the first spatial coordinates.

[0036] In some embodiments, the spatial position of the target experimental element component in the virtual experimental environment is retrieved, the first spatial coordinates adjacent to it are calculated and determined, and the generated confusing experimental element components are deployed at these first spatial coordinate positions to ensure that they can produce obvious confusing intervention and comparative identification when students perform experimental operations. Thus, by deploying confusing experimental element components, it can cover the key areas of the experimental scene, thereby improving the challenge and interactivity of the simulation experimental operation.

[0037] Through this embodiment, the complexity and interactivity of the virtual experimental environment are enhanced by using the confusing experimental element components generated by GAN, and they are set in a spatial position adjacent to the target experimental element components in the virtual experimental environment, so that students need to make deeper identification and judgment during experimental operations, thereby improving their understanding and cognitive ability of experimental elements.

[0038] In step S140, the user interaction operation is detected according to each target experiment element component and each confusing experiment element component in the virtual experiment environment to evaluate the user's ability to recognize each target experiment element information in the target experiment course.

[0039] In some embodiments, the interactive monitoring module built into the virtual experiment system monitors the students' operation behaviors during the experiment in real time, including interactive actions such as clicking, dragging, and rotating. In this way, the operation of the students on the target experimental elements and confusing experimental elements during the experiment can be analyzed, and the accuracy of their recognition of different experimental elements can be recorded and evaluated. The learning progress and degree of understanding of the students can be dynamically understood, and their mistakes in the experiment can be discovered and corrected in time. Personalized guidance based on the evaluation results can be provided to help students consolidate the key points of the experiment and improve their learning effects and experimental skills.

[0040] Figure 2 A flowchart of another example of an experimental simulation interaction method based on virtual reality technology according to an embodiment of the present application is shown.

[0041] like Figure 2 As shown, in step S210, the user interaction operation is detected according to each target experiment element component and each experiment element component in the virtual experiment environment.

[0042] For more details about step S210, please refer to the above description in combination with other examples, which will not be repeated here.

[0043] In step S220, when it is detected that the user interaction operation is a selection operation on the first obfuscated experiment element component, the first target experiment element component corresponding to the first obfuscated experiment element component is highlighted, and an incorrect operation prompt notification is generated according to the experiment element information of the first target experiment element component.

[0044] In some embodiments, sensors or input devices (such as VR handles, eye tracking, etc.) are used to monitor the user's interactive behavior in the virtual experimental environment in real time. When the user selects a certain experimental element component, the system automatically identifies whether the component is a confusing experimental element component. If the user selects the first confusing experimental element component, the system will find the first target experimental element component corresponding to the confusing component, and highlight the target experimental element component, such as using visual effects such as color change, flashing, or prominent borders, so that it is more conspicuous in the virtual environment, prompting the user to make the correct choice. At the same time, the system generates an error operation prompt notification based on the experimental element information of the first target experimental element component. The notification can be presented in the form of a pop-up window, text prompt, or voice prompt, and the content includes the correct experimental element information and the cause of the error operation, helping the user understand and correct the error. Exemplarily, the system generates a detailed error operation prompt notification based on the experimental element information of the first target experimental element component, and the prompt content can be that the user has selected the wrong experimental element, the correct experimental element information and its function, the possible impact of the error operation, etc., and can notify the user through pop-ups, voice prompts, or vibration feedback. Therefore, through highlight rendering and error operation prompt notification, users can immediately understand their own wrong operations and correct them quickly. Through this instant error prompt and correction mechanism, the user's experimental identification ability is improved, thereby reducing errors in experimental operations.

[0045] In step S230, extended sample data is generated according to the first obfuscation experiment element component and the first target experiment element component, and the extended sample data is used to perform online learning optimization on the obfuscation component generation model.

[0046] In some embodiments, based on the user's operation record, a set of extended sample data including obfuscated experimental element components and corresponding target experimental element components is generated, and the extended sample data includes obfuscated components selected incorrectly by the user and correct components highlighted by the system. The generated extended sample data is input into the obfuscated component generation model, the model is online learned and optimized, and the model parameters are iteratively updated to improve the diversity and accuracy of the obfuscated components generated by the GAN model.

[0047] More specifically, the GAN model includes a generator network and a discriminator network. The generator network processes the target experimental element components to obtain similar but different obfuscated components. The discriminator network discriminates the difference between the generated obfuscated experimental element components and the target experimental element components. Furthermore, through adversarial training, the generator network can continuously improve the generated obfuscated components to make them more realistic and confusing.

[0048] Through this embodiment, the data of the obfuscated experimental element components and the target experimental element components are integrated, and the obfuscated experimental element components that are highly confusing to the target experimental element components are used for training, and the high confusion of the target experimental element components is fully utilized. The GAN model can generate experimental element components with significant confusion effects, thereby ensuring the confusion effects of the obfuscated experimental element components generated by the model in virtual experiments, so that students pay more attention to distinguishing the obfuscated elements during the experiment, thereby more deeply understanding the characteristics of the target experimental elements in the course.

[0049] In some examples of the embodiments of the present application, when it is detected that the second target experimental element component has not been selected within a preset time period, the second target experimental element component is highlighted. More specifically, a preset time period (e.g., 2 minutes) is pre-set in the system to detect whether the user has completed the selection of each target experimental element component involved in the target experimental course within this time period. If the system detects that the user has not selected one or more of the second target experimental element components within the preset time period, a highlight rendering operation is triggered, such as border flashing, color change, transparency adjustment, etc., to prompt the user to select the missing experimental element components to ensure the integrity of the experimental elements.

[0050] In some examples of the embodiments of the present application, the obfuscation component generation model adopts a Conditional Multi-Scale Generative Adversarial Network (CMS-GAN). Figure 3 A schematic diagram of the structural connection of an example of an obfuscation component generation model according to an embodiment of the present application is shown.

[0051] like Figure 3 As shown, the generator 300 of the obfuscation component generation model includes an input layer 310 , a conditional fusion layer 320 , a multi-scale feature extraction layer 330 , a feature fusion layer 340 and a conditional generation layer 350 .

[0052] The input layer 310 is used to receive target experimental element components, obfuscation description information and experimental environment information.

[0053] Here, the obfuscated description information defines description information that is similar to the target experimental element information but has different characteristics, and the characteristics include any one of the following: physical state characteristics, morphological characteristics, color characteristics, shape characteristics, and capacity characteristics.

[0054] It should be noted that the obfuscated description information is used to guide CMS-GAN to generate obfuscated experimental element components, and to test and train the user's ability to identify the target experimental elements during the experiment. For example, the physical state characteristics are used to guide the generation of experimental element components of different physical states, such as solid, liquid, and gaseous reactants, the capacity characteristics are used to guide the generation of beaker components and test tube components of different sizes or shapes, or the color characteristics are used to guide the generation of components with different colors but similar chemical properties.

[0055] In one example of the embodiment of the present application, the characteristics can be selected in a random manner to guide the generation of the obfuscated experimental element component. In another example of the embodiment of the present application, the characteristics can be selected specifically for different types of target experimental element components to further enhance the obfuscation effect of the obfuscated experimental element component.

[0056] Figure 4 A processing flow chart of an example of a determination operation for obfuscated description information according to an embodiment of the present application is shown.

[0057] like Figure 4 As shown, in step S410, the confusion core characteristics corresponding to the target experimental element information are determined according to a preset element confusion table, and the element confusion table pre-stores multiple experimental element information and corresponding core characteristics.

[0058] In some embodiments, an element confusion table is pre-constructed, which stores multiple experimental element information and their corresponding core characteristics. The core characteristics include key attributes of the experimental elements, such as color, shape, physical state, capacity, etc. Exemplarily, a record in the element confusion table may be: "Experimental instrument-beaker-shape characteristic-capacity characteristic", "Reactant-solution-color characteristic". Thus, the core characteristics corresponding to the experimental element are matched through the element confusion table.

[0059] In step S420, an obfuscation rule template matching the obfuscation core characteristic is determined.

[0060] In some embodiments, a confusion rule template library is constructed to store multiple confusion rule templates, each template including at least one element part placeholder for replacing the corresponding part in the experimental element. Exemplarily, the confusion rule template may be "replace the color field", "replace the shape field", and "replace the capacity field".

[0061] In step S430, obfuscation description information is determined according to the obfuscation rule template and the target experiment element information.

[0062] In some implementations, according to the selected obfuscation rule template, the element part placeholder position and replacement rule in the template are parsed. For example, when the obfuscation rule template is a color replacement rule, the color placeholder is replaced with other possible colors.

[0063] Therefore, through the preset confusion rule templates and core feature replacement rules, the generated confusion description information can be efficiently and automatically generated through reasonable and precise feature replacement operations, enabling the system to dynamically generate diverse confusion experimental elements, increase the difficulty of identifying confusion experimental elements, and enhance the complexity and interactivity of the virtual experimental environment.

[0064] The conditional fusion layer 320 is used to fuse the obfuscated description information and the experimental environment information through a fully connected layer to generate a conditional vector:

[0065] , Formula (1)

[0066] In the formula, and are the weights and biases of the conditional fusion layer, is the activation function; Indicates confusing description information. Indicates the experimental environment information. represents the conditional vector.

[0067] Here, the obfuscation description information and experimental environment variables are fused as conditional inputs, such as beakers of different capacities, reactants of different physical states, and reactants of different colors. This information can help the generator generate a variety of obfuscation components, and the experimental environment variables, such as the lighting and occlusion information of the experimental site, can update the display effect of the generated experimental elements in the virtual environment. Therefore, by fusing the conditional information with the model data through the fusion layer, the generation strategy can be dynamically adjusted during the generation process to ensure that the obfuscated experimental element components that meet the conditions can be generated.

[0068] The multi-scale feature extraction layer 330 is used to extract the scale features of the three-dimensional model data of the target experimental element component at various scales through multiple three-dimensional convolution layers:

[0069] , Formula (2)

[0070] In the formula, and They are The weights and biases of the convolutional layers, represents the convolution operation, Indicates The scale features output by the three-dimensional convolutional layer, each of which has a convolution kernel of the corresponding scale.

[0071] By introducing multi-level convolution operations in the generative network, features of different scales can be extracted separately. For example, a larger-scale convolution layer can capture the overall morphological features, and a smaller-scale convolution layer can capture the detailed features. Multi-scale convolution layers are added to the generative network to extract features of different scales layer by layer. Each convolution operation is followed by a pooling layer to reduce the size of the feature map while retaining important features.

[0072] The feature fusion layer 340 is used to fuse the feature maps of each scale through skip connections to obtain fused multi-scale features:

[0073] , Formula (3)

[0074] In the formula, represents the total number of 3D convolutional layers, Represents the fusion of multi-scale features.

[0075] In some implementations, feature maps of different scales may be connected by using skip connections to fuse feature maps of different scales to obtain fused multi-scale features.

[0076] The conditional generation layer 350 is used to combine the fused multi-scale features and the conditional vector, and generate the confusion experiment element component through the fully connected layer:

[0077] , Formula (4)

[0078] In the formula, Indicates the obfuscated experimental element component, and They represent the weight matrix and bias term of the conditional generation layer respectively.

[0079] Through this embodiment, by adopting CMS-GAN, conditional GAN ​​and multi-scale feature extraction technology, the generator can flexibly adjust the generation strategy according to the different confusion description information and experimental environment variables of the input, and the generated confusion experiment element components are closer to the real experimental elements in morphology and characteristics, so that the confusion experiment element components are not only diverse, but also can generate a variety of different confusion components according to different confusion description information and experimental environment variables (such as lighting conditions), so as to improve the authenticity and richness of the experimental simulation, for example, it can match the condition requirements of the virtual experimental environment. In addition, through CMS-GAN, the generated confusion experiment element components can effectively simulate the confusion factors in the real experiment in the virtual experimental environment, helping users to improve the ability to identify the experimental elements when performing experimental operations. By frequently contacting different types of confusion components, students can more accurately identify and operate the target experimental elements, and improve their learning effect on the experimental elements in the experimental course.

[0080] Therefore, by adopting the conditional multi-scale generative adversarial network, the realism, diversity and adaptability of the system can be significantly improved in the virtual reality experiment simulation interaction method, the user's recognition ability and learning effect can be enhanced, and the course-adaptive personalized experimental training scenario and comprehensive learning effect evaluation can be provided.

[0081] Regarding the loss function of CMS-GAN, in some examples of the embodiments of the present application, the loss function of the generator of CMS-GAN is improved to realize a comprehensive loss function based on perceptual loss. Perceptual loss not only focuses on the difference at the pixel level, but also pays more attention to the similarity of high-level features, so it can generate more natural and realistic modeling results.

[0082] In some examples of the embodiments of the present application, the loss function of the generator of the obfuscated component generation model is for:

[0083] , Formula (5)

[0084] , Formula (6)

[0085] , Formula (7)

[0086] In the formula, represents the adversarial loss term, represents the perceptual loss term, and represents the loss term weight hyperparameter; Indicates the noise expectations, noise From the noise distribution ; Representation Discriminator To generate samples and conditions The logarithm of the output of , indicating the probability that the generated sample is identified as a real sample; Represents the discriminator for the real sample and conditions judgment; Represents a noise-based generator and conditions The generated samples, is the data distribution, which represents the distribution of real data; Indicates the sample The expected value of the data distribution, represents the first The feature mapping function of the layer, represents the L2 norm, Represents the number of network layers of the pre-trained neural network; the pre-trained neural network is used to extract high-level features of generated samples and real samples and compare them in the perceptual loss.

[0087] It should be noted that the types of pre-trained neural networks can be diverse, such as VGG, ResNet, Inception, etc., which generate high-level features of samples and target samples by calculation and compare them in perceptual loss.

[0088] For example, a VGG network can be used, which consists of multiple convolutional layers and pooling layers, has a simple structure and powerful feature extraction capabilities. More specifically, VGG16 or VGG19 can be used, which have 16 or 19 weight layers, respectively.

[0089] Exemplarily, in the perceptual loss calculation, the outputs of several convolutional layers of the VGG network can be used as feature maps. Specifically, the low-level features, middle-level features, and high-level features corresponding to the input are extracted. More specifically, the first few convolutional layers of the VGG network are selected to output low-level features, including low-level information such as the edge and texture of the image, the middle few convolutional layers of the VGG network are selected to output the shape, structure and other medium-level information of the image, and the last few convolutional layers of the VGG network are selected to output high-level features such as semantic information and abstract features of the image. Selecting features at multiple levels can comprehensively measure the similarity between the generated samples and the target samples. Thus, through perceptual loss, the generated samples are more similar to the target samples in the high-dimensional feature space, which enhances the fidelity and consistency of the generated samples and makes the simulation experiment closer to the actual experimental scenario.

[0090] It should be noted that the complete loss function of the discriminator of CMS-GAN remains unchanged, and its function is still to distinguish the authenticity of real samples and generated samples. More specifically, the loss function of the discriminator of the confusion component generation model is for:

[0091] , formula (8)

[0092] In the formula, Represents the real sample Expectations, real samples From .

[0093] It should be noted that the perceptual loss makes the generated samples more similar to the target samples in the high-dimensional feature space. The perceptual loss uses the multi-layer feature mapping of the pre-trained neural network to ensure that the generated experimental element components are close to the real experimental elements in both visual and semantic aspects. As a result, the generated experimental element components are not only realistic in appearance, but also consistent with the real samples in high-level semantic information. The generated experimental elements are not only visually realistic, but also highly consistent with the real experimental elements in the feature space, which enhances the realism of the virtual experiment simulation. In addition, the perceptual loss ensures that the generated confusing experimental elements retain the core features of the target experimental elements in perception. By generating confusing experimental elements with high recognition, students can effectively train and improve their recognition ability in experimental operations, thereby deepening their understanding of the experimental principles and steps. For example, students cannot only know that there is a beaker in the experimental element, but also need to know what shape or capacity the beaker should be.

[0094] Through this embodiment, by adding perceptual loss on the basis of the basic loss function of CMS-GAN and using the multi-layer feature mapping of the pre-trained neural network, the performance and educational effect of the confusing experimental element generation model can be significantly improved, including improving the perceptual similarity of the generated samples, enhancing the recognition of the generated confusing experimental elements, optimizing the model performance, and providing a personalized learning experience. As a result, not only the realism and diversity of virtual experimental simulation are enhanced, but also the students' in-depth understanding of the course content is effectively promoted, providing a more efficient solution for smart education.

[0095] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of actions combined, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0096] Figure 5 A structural block diagram of an example of an experimental simulation interaction system based on virtual reality technology according to an embodiment of the present application is shown.

[0097] like Figure 5 As shown, the experimental simulation interaction system 500 based on virtual reality technology includes a target element parsing unit 510, a confusing element generating unit 520, a confusing element setting unit 530 and an element interaction detecting unit 540.

[0098] The target element parsing unit 510 is used to parse multiple target experiment element information in the target experiment course; the experiment element includes any one of the following: experimental instruments, reactants and catalysts.

[0099] The confusion element generation unit 520 is used to determine a matching target experiment element component from the virtual experiment environment according to each target experiment element information, and input the target experiment element component into the confusion component generation model to obtain a confusion experiment element component corresponding to the target experiment element component; the experiment element information corresponding to the confusion experiment element component is different from the target experiment element information; multiple experiment element components are deployed in the virtual experiment environment, and each of the experiment element components is deployed at a corresponding spatial position; the confusion component generation model adopts a generative adversarial network.

[0100] The obfuscation element setting unit 530 is used to respectively determine each first spatial coordinate adjacent to the spatial position of each target experiment element component, and set the obfuscation experiment element component in the virtual experiment environment according to each first spatial coordinate.

[0101] The element interaction detection unit 540 is used to detect user interaction operations according to each of the target experiment element components and each of the confusing experiment element components in the virtual experiment environment to evaluate the user's ability to recognize each of the target experiment element information in the target experiment course.

[0102] In some embodiments, an embodiment of the present application provides a non-volatile computer-readable storage medium, in which one or more programs including execution instructions are stored. The execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the steps of any of the above-mentioned experimental simulation interaction methods based on virtual reality technology in the present application.

[0103] In some embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes any step of the above-mentioned experimental simulation interaction method based on virtual reality technology.

[0104] In some embodiments, the embodiments of the present application also provide an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the experimental simulation interaction method based on virtual reality technology.

[0105] Figure 6 FIG. 1 is a schematic diagram of the hardware structure of an electronic device for executing an experimental simulation interaction method based on virtual reality technology provided by another embodiment of the present application. Figure 6 As shown, the device includes:

[0106] One or more processors 610 and memory 620, Figure 6 A processor 610 is taken as an example.

[0107] The device for executing the experimental simulation interaction method based on virtual reality technology may further include: an input device 630 and an output device 640 .

[0108] The processor 610, the memory 620, the input device 630 and the output device 640 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0109] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the experimental simulation interaction method based on virtual reality technology in the embodiment of the present application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 620, that is, realizing the experimental simulation interaction method based on virtual reality technology in the above method embodiment.

[0110] The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include a memory remotely arranged relative to the processor 610, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0111] The input device 630 can receive input digital or character information and generate signals related to user settings and function control of the electronic device. The output device 640 can include a display device such as a display screen.

[0112] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, the experimental simulation interaction method based on virtual reality technology in any of the above method embodiments is executed.

[0113] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0114] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:

[0115] (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0116] (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.

[0117] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0118] (4) Other onboard electronic devices with data interaction functions, such as on-board devices installed in vehicles.

[0119] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An experimental simulation interaction method based on virtual reality technology, comprising: Analyze the information of multiple target experimental elements in the target experimental course; the experimental elements include any one of the following: experimental instruments, reactants and catalysts; For each of the target experimental element information, a matching target experimental element component is determined from the virtual experimental environment according to the target experimental element information, and the target experimental element component is input into the obfuscation component generation model to obtain an obfuscation experimental element component corresponding to the target experimental element component; the experimental element information corresponding to the obfuscation experimental element component is different from the target experimental element information; a plurality of experimental element components are deployed in the virtual experimental environment, and each of the experimental element components is respectively deployed at a corresponding spatial position; the obfuscation component generation model adopts a generative adversarial network; Respectively determine each first spatial coordinate adjacent to the spatial position of each target experimental element component, and set the obfuscated experimental element component in the virtual experimental environment according to each first spatial coordinate; According to each of the target experiment element components and each of the confusing experiment element components in the virtual experiment environment, detecting user interaction operations to evaluate the user's ability to recognize each of the target experiment element information in the target experiment course; The obfuscation component generation model adopts a conditional multi-scale generative adversarial network; the generator of the obfuscation component generation model includes an input layer, a conditional fusion layer, a multi-scale feature extraction layer, a feature fusion layer and a conditional generation layer; The input layer is used to receive target experimental element components, obfuscated description information and experimental environment information; the obfuscated description information defines description information similar to the target experimental element information but with different characteristics, and the characteristics include any one of the following: physical state characteristics, morphological characteristics, color characteristics, shape characteristics and capacity characteristics; The conditional fusion layer is used to fuse the confusion description information and the experimental environment information through the fully connected layer to generate a conditional vector: ， In the formula, and are the weights and biases of the conditional fusion layer, is the activation function; Indicates confusing description information. Indicates the experimental environment information. represents the conditional vector; The multi-scale feature extraction layer is used to extract the scale features of the three-dimensional model data of the target experimental element component at various scales through multiple three-dimensional convolutional layers: ， In the formula, and They are The weights and biases of the convolutional layers, represents the convolution operation, Indicates The scale features output by the three-dimensional convolutional layer, each of which has a convolution kernel of the corresponding scale; The feature fusion layer is used to fuse the feature maps of each scale through jump connections to obtain fused multi-scale features: ， In the formula, represents the total number of 3D convolutional layers, Indicates the fusion of multi-scale features; The conditional generation layer is used to combine the fused multi-scale features and the conditional vector, and generate the confusion experiment element component through the fully connected layer: ， In the formula, Indicates the obfuscated experimental element component, and They represent the weight matrix and bias term of the conditional generation layer respectively.

2. The method according to claim 1, wherein: After detecting user interaction operations according to each of the target experiment element components and each of the experiment element components in the virtual experiment environment, the method further includes: When it is detected that the user interaction operation is a selection operation on a first obfuscated experiment element component, a first target experiment element component corresponding to the first obfuscated experiment element component is highlighted, and an incorrect operation prompt notification is generated according to the experiment element information of the first target experiment element component.

3. The method according to claim 2, wherein: After generating an incorrect operation prompt notification according to the experimental element information of the first target experimental element component, the method further includes: According to the first obfuscation experiment element component and the first target experiment element component, expanded sample data is generated, and the expanded sample data is used to perform online learning optimization on the obfuscation component generation model.

4. The method according to claim 1, wherein: After detecting user interaction operations according to each of the target experiment element components and each of the experiment element components in the virtual experiment environment, the method further includes: When it is detected that the second target experiment element component has not been selected within a preset time period, highlight rendering is performed on the second target experiment element component.

5. The method according to claim 4, wherein: The determination operation for the obfuscated description information includes: Determining the obfuscated core characteristics corresponding to the target experimental element information according to a preset element confusion table; the element confusion table pre-stores a plurality of experimental element information and corresponding core characteristics; Determine an obfuscation rule template that matches the obfuscation core characteristic; the obfuscation rule template includes at least one element part placeholder, and each of the element part placeholders is used to replace a corresponding part in the experimental element; Obfuscation description information is determined according to the obfuscation rule template and the target experimental element information.

6. The method according to claim 4, wherein: The loss function of the generator of the obfuscated component generation model for: ， ， ， In the formula, represents the adversarial loss term, represents the perceptual loss term, and represents the loss term weight hyperparameter; Indicates the noise expectations, noise From the noise distribution ; Representation Discriminator To generate samples and conditions The logarithm of the output of , indicating the probability that the generated sample is identified as a real sample; Represents the discriminator for the real sample and conditions judgment; Represents a noise-based generator and conditions The generated samples, is the data distribution, which represents the distribution of real data; Indicates the sample The expected value of the data distribution, represents the first The feature mapping function of the layer, represents the L2 norm, Represents the number of network layers of a pre-trained neural network; the pre-trained neural network is used to extract high-level features of generated samples and real samples and compare them in the perceptual loss.

7. An experimental simulation interaction system based on virtual reality technology, used to implement the experimental simulation interaction method based on virtual reality technology as claimed in claim 1; the system comprises: A target element parsing unit, used for parsing information of multiple target experiment elements in a target experiment course; Experimental elements include any of the following: experimental apparatus, reactants and catalysts; A confusion element generation unit is used to determine a matching target experiment element component from a virtual experiment environment according to each target experiment element information, and input the target experiment element component into a confusion component generation model to obtain a confusion experiment element component corresponding to the target experiment element component; the experiment element information corresponding to the confusion experiment element component is different from the target experiment element information; a plurality of experiment element components are deployed in the virtual experiment environment, and each of the experiment element components is respectively deployed at a corresponding spatial position; the confusion component generation model adopts a generative adversarial network; A confusion element setting unit, used to respectively determine each first spatial coordinate adjacent to the spatial position of each target experimental element component, and set the confusion experimental element component in the virtual experimental environment according to each first spatial coordinate; The element interaction detection unit is used to detect user interaction operations according to each of the target experiment element components and each of the confusing experiment element components in the virtual experiment environment to evaluate the user's ability to recognize each of the target experiment element information in the target experiment course.

Images