Education VR simulation scene construction method based on artificial intelligence

Abstract

The invention discloses a method for constructing an education VR simulation scene based on artificial intelligence, and the method comprises the steps: obtaining real image data, carrying out the pixel-level fusion of different resolutions and waveband data through a convolutional neural network, obtaining a multivariate fusion image, constructing a GAN model containing a generator and a discriminator, generating a first VR scene through adversarial training, and carrying out the recognition of the first VR scene. The method comprises the following steps: acquiring a first VR scene, segmenting pixels of the first VR scene to determine a region type, filling VR elements by using a variational auto-encoder to obtain a second VR scene, mapping and associating a teaching knowledge graph with the second VR scene to form a teaching path, constructing a recurrent neural network to learn emotion and view transition states of students on the teaching path, and updating the second VR scene according to preferences. Therefore, a final VR simulation scene is obtained for teaching. According to the method, a highly-simulated VR scene with teaching pertinence can be generated, the student state can be accurately perceived, the teaching immersion and effect are improved, and meanwhile, the method has relatively good interpretability.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a method for constructing an educational VR simulation scene based on artificial intelligence. Background Art

[0002] With the rapid development of information technology, the application potential of virtual reality (VR) technology in the field of education is becoming increasingly prominent. The traditional education model is often limited by the limited teaching resources, the single teaching method, and the lack of student participation and experience. For example, in high school geography teaching, it is difficult for students to intuitively experience abstract geographical concepts, complex geographical processes, and geographical features of remote areas, which leads to certain difficulties in understanding and mastering knowledge.

[0003] At the same time, artificial intelligence technology has made great breakthroughs in recent years, showing excellent performance in image recognition, natural language processing, data mining and other fields. Introducing artificial intelligence technology into VR education scene construction can effectively solve many challenges faced by traditional VR scene construction. For example, in terms of scene generation, traditional VR scene construction often relies on manual modeling, which not only consumes a lot of time and manpower, but also the generated scenes may lack diversity and realism.

[0004] The artificial intelligence-based educational VR simulation scene construction method uses the image analysis and data processing capabilities of artificial intelligence to optimize VR scene generation, making it more realistic and diverse, enhancing immersion and comfort, and ultimately creating a novel and efficient teaching model, giving students a high-quality learning experience to adapt to new trends in educational development and help students grow comprehensively. Summary of the invention

[0005] The purpose of the present invention is to provide a method for constructing an educational VR simulation scene based on artificial intelligence.

[0006] To achieve the above object, the present invention is implemented according to the following technical solutions:

[0007] The first aspect of the present invention provides a method for constructing an educational VR simulation scene based on artificial intelligence, comprising:

[0008] Acquire real image data, and fuse real image data of different resolutions and bands based on the pixel-level fusion method of convolutional neural network to obtain a multi-dimensional fused image;

[0009] Constructing a GAN model including a generator network and a discriminator network, and using the generator network and the discriminator network to perform adversarial training based on the multivariate fusion image to obtain a first VR scene;

[0010] Apply pixel segmentation to the first VR scene to identify different region types, and fill VR elements into the regions through a variational autoencoder according to semantic information of different regions to obtain a second VR scene;

[0011] Mapping and associating the preset teaching knowledge graph with the second VR scene to obtain a teaching path;

[0012] A recurrent neural network is constructed to learn the students' emotional state and perspective transfer state on the teaching path, and the second VR scene is updated based on the learned preference to obtain a final VR simulation scene.

[0013] As a further method, the pixel-level fusion method based on convolutional neural network, a method for fusing real image data of different resolutions and different bands, includes:

[0014] The U-Net architecture is selected to build a convolutional neural network, and images of different resolutions and bands are input into channels of different convolutional layers to extract image features, including texture features and shape features;

[0015] Assign weights proportional to the resolution to the image features. The middle layer of the convolutional neural network performs weighted summation based on the weights of the image features to obtain a fused feature map.

[0016] The fused feature map is upsampled and reconstructed to generate a multivariate fused image.

[0017] As a further method, the method of using a generator network and a discriminator network to perform adversarial training based on the multivariate fusion image includes:

[0018] The multi-dimensional fusion image is input into the generator network, and the image features are transformed and information is reorganized using a structure including a convolutional layer, a fully connected layer, and a ReLU activation function layer to generate an initial VR scene image.

[0019] The initial VR scene image and the multi-element fusion image are input into the discriminator network, and features are extracted through the convolution layer and classified and judged through the fully connected layer, and the probability value of the output image is the real image;

[0020] The loss function is constructed based on the probability value output by the discriminator network, and the expression is:

[0021]

[0022] Where m is the number of multivariate fusion images, z i is the i-th multivariate fusion image, G(z i ) is the generator network based on z i The generated image, D(G(z i )) is the discriminator network for G(zi ), n is the probability value of G(z i ), the number of features in x j is the jth feature point, is the discriminator network for G(z i ) in x j The gradient at

[0023] The convergence condition is set to that the change in the loss function is less than 0.001 in 100 consecutive iterations, and iterative training is performed. In each iteration, the parameters in the generator network and the discriminator network are updated by back propagation.

[0024] As a further method, the method of applying pixel segmentation to the first VR scene to identify different area types includes:

[0025] Obtain edge information of the first VR scene using the Canny edge detection algorithm, and obtain multiple spatial regions based on the edge information;

[0026] The superpixel similarity between spatial regions is calculated, and the spatial regions with superpixel similarity greater than 0.7 are regarded as the same spatial region type. The calculation formula for superpixel similarity is:

[0027]

[0028] Among them, S i and S j are the superpixel sets corresponding to the two spatial regions, W 1 is the diagonal weight matrix of color features, C i and C j S i and S j The corresponding color feature vector, σ 1 is the color similarity attenuation speed control coefficient, W 2 is the diagonal weight matrix of the position feature, L i and L j S i and S j The corresponding position eigenvector, σ 2 is the control coefficient of the position similarity decay speed, D ij For S i and S j The texture feature difference tensor, ‖D ij ‖ F D ij The Frobenius norm, σ 3 is the control coefficient of texture similarity decay speed.

[0029] As a further method, the method of filling VR elements for regions by using a variational autoencoder according to semantic information of different regions includes:

[0030] Analyze the characteristics of each spatial area, determine the semantic information, collect VR element sample data related to the semantic information, and perform preprocessing, including cropping, scaling, and normalization operations;

[0031] Construct a variational autoencoder model, including an encoder and a decoder, where the encoder extracts the features of the VR element sample data and maps them to the mean vector and variance vector to define the distribution of the latent space, expressed as:

[0032]

[0033] Among them, μ is the mean vector, σ 2 is the variance vector, Encoder is the encoder for encoding operation, X is the input data, L is the total number of encoder layers, w l and b l are the weight and bias of the lth layer, ξ is the ReLU activation function, BN means batch normalization operation, K l is the feature map of the lth layer, w k,l and b k,l are the weight and bias of the kth feature map in the lth layer, X k is the input of the kth feature map;

[0034] The variational autoencoder parameters are optimized according to the loss function, where the loss function consists of the reconstruction error and the KL divergence, and the expression is:

[0035]

[0036] Where N is the number of samples in the training data set, H, W and C are the height, width and number of channels of the image respectively, and x i,h,w,c is the original pixel value of the i-th sample image at the (h, w, c) position, is the pixel value of the i-th sample image at the (h, w, c) position generated by the decoder, z is the dimension of the latent space, μ j and σ j are the mean and variance of the jth dimension in the latent space, respectively. β is the balance coefficient, which is used to balance the relative importance of reconstruction loss and KL divergence in the total loss function and is set to 0.5.

[0037] After the decoder obtains the vector from the latent space sampling, it restores the VR elements of the original size. For dynamic VR elements, it adds dynamic effects on the basis of restoration based on the recorded motion trajectory information, and places the generated VR elements in the corresponding space area according to the actual layout and proportion.

[0038] As a further method, the method of mapping and associating the preset teaching knowledge graph with the second VR scene includes: analyzing the preset teaching knowledge graph structure to determine the knowledge node hierarchy and association, identifying the key elements corresponding to the knowledge graph nodes in the second VR scene, and combining the key elements in the second VR scene according to the logical relationship of the knowledge graph to form a teaching path.

[0039] As a further method, the method of constructing a recurrent neural network to learn the emotional state and perspective transfer state of the students on the teaching path includes:

[0040] Collect students' multimodal data on the teaching path, including behavioral data, facial data, and physiological data, input them into the pre-trained emotion classification model, output the probability distribution of the students' current emotional state, and select the emotional state with the highest probability as the emotional state prediction result;

[0041] A recurrent neural network model was constructed using a long short-term memory network. The multimodal data and the emotion state prediction results were combined as a training set to train the model. A softmax function was added to the output layer of the model to output the probability distribution of each viewing area. The expression of the probability distribution of the viewing area is:

[0042]

[0043] Among them, y t,j is the probability that the student's next perspective is in the jth region at time t, w j and w k are the weight vectors of the output layer corresponding to the j-th viewing area and the k-th viewing area, respectively. T is the transposition operation, o t,L is the output value of the last layer after the recurrent neural network is calculated at time t, c t,L is the cell state value of the last layer after the recurrent neural network calculation, b j and b k is the bias of the output layer corresponding to the j-th viewing area and the k-th viewing area, n is the total number of viewing areas divided in the VR scene, where o t,L and c t,L The calculation formula is:

[0044] o t,L =σ(W o，l ·[h t-1,l ;X t ]+b o，l )

[0045]

[0046] Among them, W o,l , Wf,l , W i,l and W c,l are the weight matrices of the output gate, forget gate, input gate, and cell state update in layer l, respectively. o,l , b f,l , b i,l and b c,l and are the bias vectors of the output gate, forget gate, input gate and cell state update at layer l, respectively. R is the number of hidden layers. δ is the sigmoid activation function. h t-1,l is the hidden state of the lth layer at the previous moment t-1, X t is the eigenvector at time t, c t-1,L is the cell state value of the last layer of the recurrent neural network at time t-1.

[0047] As a further method, the method for updating the second VR scene based on the learned preference includes:

[0048] Judge the predicted emotional state output by the recurrent neural network. If the emotional state is positive, keep the current scene. If the emotional state is negative, adjust the scene color and layout.

[0049] According to the probability distribution of the viewing area output by the recurrent neural network model, the position of the VR element is adjusted. The expression is:

[0050]

[0051]

[0052] Among them, (x i ,y i ,z i ) is the original coordinate of the VR element, is the coordinates adjusted for VR elements, P j is the probability distribution value of the viewing area, θ h and θ v are the horizontal and vertical angles of the students’ perspective preferences, q is the number of VR elements, and e is the number of VR elements. i is the i-th VR element, A(e i ) is e i The distance from the current view center.

[0053] In a second aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes the method steps described in the first aspect.

[0054] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device executes the method steps described in the first aspect.

[0055] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects:

[0056] (1) The present invention integrates real image data of different resolutions and bands through the pixel-level fusion method of convolutional neural network, so that the generated multi-dimensional fusion image can combine the advantages of images from different sources, and both the presentation of macro-landforms and the display of micro-details are richer and more accurate.

[0057] (2) The present invention further enhances the authenticity of the scene through adversarial training of the GAN model. The generator network can learn the complex features and patterns in real images. The generated VR scene is highly similar to the real scene in terms of three-dimensional sense, light and shadow effects, etc., which greatly enhances the sense of immersion and helps to improve students' learning interest and concentration.

[0058] (3) The present invention fills VR elements in regions through pixel segmentation combined with variational autoencoders, thereby intelligently customizing VR scene content according to semantic information of different regions, making the scenes more diverse and meeting the specific needs of teaching content, and better assisting students in understanding specific knowledge points.

[0059] (4) The present invention can capture students' reactions in the learning process in real time by learning students' emotional states and perspective shift states through a recurrent neural network, and can adjust the scenes in a targeted manner to adapt to students' learning preferences, thereby helping to improve students' learning outcomes and knowledge retention rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 The present invention is a flowchart of a method for constructing an educational VR simulation scene based on artificial intelligence in an embodiment of the present invention.

[0061] Figure 2 It is a schematic diagram of the structure of an electronic device in an embodiment of this specification. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0063] Reference Figure 1 As shown, the present invention provides a method for constructing an educational VR simulation scene based on artificial intelligence, comprising:

[0064] S100 obtains real image data and fuses real image data of different resolutions and bands based on the pixel-level fusion method of convolutional neural network to obtain a multi-dimensional fused image;

[0065] In the actual evaluation, the invention was applied to the creation of a VR scene of "karst landform" in high school geography. Specifically, satellite remote sensing images related to karst landforms were obtained from a professional geographic image database. The image resolution ranged from 0.5 meters to 2 meters, covering the red, green, blue bands of visible light and the near-infrared band. The total amount of data was about 300GB. The high-resolution visible light band images were input into the shallow convolutional layers (1st to 3rd layers) of the network to capture the subtle texture and edge features of the karst landforms. The medium-resolution near-infrared band images were input into the middle convolutional layers (4th to 6th layers) to extract feature information related to vegetation coverage, soil moisture and groundwater level. The low-resolution multispectral synthetic images were input into the deep convolutional layers (7th to 9th layers) to obtain the macroscopic morphology and regional distribution characteristics of the landforms.

[0066] In the actual evaluation, weights proportional to the resolution are assigned to image features. Specifically, the weight of high-resolution image features is determined to be 0.5, the weight of medium-resolution image features is 0.3, and the weight of low-resolution image features is 0.2. The middle layer of the convolutional neural network performs weighted summation based on the weights of the image features to obtain a fused feature map, which is then upsampled and reconstructed to generate a multivariate fused image.

[0067] S200 constructs a GAN model including a generator network and a discriminator network, and uses the generator network and the discriminator network to perform adversarial training based on the multivariate fusion image to obtain a first VR scene;

[0068] In the actual evaluation, the generator network uses a structure consisting of 5 convolutional layers, 2 fully connected layers and a ReLU activation function layer to transform the features of the image and reorganize the information to generate the initial VR scene image. The convolution kernel sizes of the convolutional layers are 7×7, 5×5, 3×3, 3×3, and 3×3, respectively, and the step sizes are 2, 2, 1, 1, and 1, respectively. There are 2 fully connected layers with 1024 and 512 nodes, respectively. The discriminator network extracts features through 4 convolutional layers (convolution kernel size 5×5, step size 2) and 2 fully connected layers for classification and judgment. The output image is a real image. Probability value, the multivariate fusion image is input into the generator network, and its structure is used to generate the initial VR scene image, and iterative training is performed. In each iteration, the parameters in the generator network and the discriminator network are updated by back propagation. After 6000 iterative training, the VR scene image generated by the generator network has a similarity of about 83% with the real image, and can better present the three-dimensional and realistic sense of karst landform in terms of visual effect. The error of the height and spacing ratio of the peak forest is controlled within 8%, and the shape and size of the cave are consistent with the actual situation at about 85%.

[0069] S300 applies pixel segmentation to the first VR scene to identify different region types, and fills VR elements into the regions through a variational autoencoder according to semantic information of different regions, to obtain a second VR scene;

[0070] In the actual evaluation, when the segmentation test was conducted on the complex karst landform VR scene containing peak forests, caves and surface water systems, the accuracy of regional division reached more than 90%, and the peak forest area could be accurately distinguished from the surrounding vegetation and soil areas. The segmentation accuracy of different caves and passages inside the cave was about 85%, and the accuracy of the boundary division between the surface water system and the surrounding land was about 92%. The characteristics of each spatial area were analyzed to determine the semantic information. For the peak forest area, its semantic information was determined to be limestone peaks. VR element sample data related to limestone peaks, such as limestone models of different shapes and textures, were collected, cropped, scaled and normalized, and a variational autoencoder model was constructed, including an encoder and a decoder. The encoder extracts the features of the VR element sample data and maps them to the mean vector and variance vector to define the distribution of the latent space. After the decoder obtains the vector from the latent space sampling, it restores the VR elements of the original size. For dynamic VR elements, such as water droplets falling in the cave and water flow in the underground river, dynamic effects are added on the basis of restoration according to the recorded motion trajectory information, and the generated VR elements are placed in the corresponding space area according to the actual layout and proportion. The second VR scene finally generated has a significant improvement in the richness and realism of karst landform elements. The rock texture of the peak forest is more delicate and realistic, the stalactites and stalagmites in the cave are more diverse and natural, and the water flow effect of the underground river is smoother and more vivid.

[0071] S400 maps and associates the preset teaching knowledge graph with the second VR scene to obtain a teaching path;

[0072] It needs to be explained that this step realizes the precise positioning of educational content for VR scenes. The knowledge graph covers a systematic and comprehensive knowledge structure under a specific subject or theme. By mapping and associating it with the second VR scene, it can determine the elements and areas that should be highlighted and highlighted in the scene based on the key points of knowledge. For example, in the construction of VR scenes of mathematical geometry, according to the knowledge nodes such as the properties and theorems of solid geometric figures in the knowledge graph, the relationship between the vertices, edges, and faces of the cube is accurately positioned and strengthened in the scene, so that the VR scene is closely centered around the core of teaching. Secondly, it promotes the structuring and hierarchical presentation of knowledge. Through mapping associations, the elements in the VR scene are arranged and combined in an orderly manner according to the logical order of the knowledge graph, forming a natural and smooth teaching path;

[0073] In the actual evaluation, the high school geography "karst landform" preset teaching knowledge map, knowledge nodes include the formation causes of karst landform (the composition and characteristics of limestone, the principle of water dissolution), landform characteristics (the form and structure of peak forests, peak clusters, caves, and underground rivers), distribution areas (the main karst landform distribution areas in the world and their geographical environment characteristics) and the impact on human activities (tourism development, agricultural production, water resources utilization, etc.). The node level is divided into the basic concept layer (such as the chemical composition of limestone), the principle layer (the dissolution process of water on limestone) and the application layer (tourism resource development strategy in karst areas). The nodes are related through causal relationships, inclusion relationships, etc. to identify the second VR field The peak forest in the second VR scene corresponds to the karst landform feature node in the knowledge graph, and the stalactite formation process in the cave corresponds to the dissolution principle node in the knowledge graph. According to the logical relationship of the knowledge graph, the key elements are combined in the second VR scene to form a teaching path, specifically: starting with the composition and characteristics of limestone, showing how it gradually forms caves, peak forests and other landforms under the dissolution of water, then introducing the global distribution of karst landforms and their relationship with local climate, geology and other factors, and finally discussing the impact of karst landforms on local human activities (such as tourism, agriculture, water conservancy, etc.) and how humans can rationally utilize and protect karst landform resources.

[0074] S500 constructs a recurrent neural network to learn the student's emotional state and perspective transfer state on the teaching path, and updates the second VR scene based on the learned preference to obtain a final VR simulation scene.

[0075] In the actual evaluation, multimodal data of 80 high school students in the learning process according to the above teaching path were collected, including behavioral data (movement trajectory in VR scene, perspective switching frequency, number of operations on interactive elements), facial data (facial expressions collected by the camera on the head-mounted device, and expression characteristics such as interest, confusion, and concentration were analyzed) and physiological data (heart rate changes, collected by the bracelet device). These data were input into the pre-trained emotion classification model, and the probability distribution of the student’s current emotional state was output. The emotional state with the highest probability was selected as the emotional state prediction result. Among them, when student A was observing the complex internal structure of the cave, if his facial expression showed confusion, his heart rate increased slightly, he stayed in the area for a long time and his operation interaction decreased, the emotion classification model could accurately judge his emotional state as negative;

[0076] In the actual evaluation, a recurrent neural network model was constructed with a long short-term memory network. The multimodal data and the prediction results of the emotional state were combined as a training set to train the model. The hidden layer of the model was set to 3 layers, and the softmax function was added to the output layer to output the probability distribution of each viewing area. In the test of the viewing area prediction of students observing the peak forest area in the VR scene, the prediction accuracy reached about 76%. The emotional state prediction results output by the recurrent neural network were judged. For the scene adjustment driven by the emotional state, specifically: when the student's emotional state is negative, the color saturation of the peak forest area is reduced by 30%, making the overall tone colder to relieve visual fatigue and create a tranquil atmosphere; at the same time, some minor vegetation elements are simplified, their number and degree of detail are reduced, and the main features of the karst landform are highlighted. For the element adjustment guided by the viewing area, specifically: when the student's perspective is more focused on the cave area, the position of the stalactites and stalagmites in the cave is adjusted to make it closer to the center of the perspective and the distribution is more in line with the visual habits. At the same time, these elements are enlarged by 20% to enhance their visual impact, so that students can better observe and understand their morphological characteristics and formation process.

[0077] In this embodiment, the pixel-level fusion method based on the convolutional neural network is a method for fusing real image data of different resolutions and different bands, including:

[0078] The U-Net architecture is selected to build a convolutional neural network, and images of different resolutions and bands are input into channels of different convolutional layers to extract image features, including texture features and shape features;

[0079] Assign weights proportional to the resolution to the image features. The middle layer of the convolutional neural network performs weighted summation based on the weights of the image features to obtain a fused feature map.

[0080] The fused feature map is upsampled and reconstructed to generate a multivariate fused image.

[0081] In this embodiment, the method of performing adversarial training based on the multivariate fusion image using a generator network and a discriminator network includes:

[0082] The multi-dimensional fusion image is input into the generator network, and the image features are transformed and information is reorganized using a structure including a convolutional layer, a fully connected layer, and a ReLU activation function layer to generate an initial VR scene image.

[0083] The initial VR scene image and the multi-element fusion image are input into the discriminator network, and features are extracted through the convolution layer and classified and judged through the fully connected layer, and the probability value of the output image is the real image;

[0084] The loss function is constructed based on the probability value output by the discriminator network, and the expression is:

[0085]

[0086] Where m is the number of multivariate fusion images, z i is the i-th multivariate fusion image, G(z i ) is the generator network based on z i The generated image, D(G(z i )) is the discriminator network for G(z i ), n is the probability value of G(z i ), the number of features in x j is the jth feature point, is the discriminator network for G(z i ) in x j The gradient at

[0087] The convergence condition is set to that the change in the loss function is less than 0.001 in 100 consecutive iterations, and iterative training is performed. In each iteration, the parameters in the generator network and the discriminator network are updated by back propagation.

[0088] In this embodiment, the method of applying pixel segmentation to the first VR scene to identify different area types includes:

[0089] Obtain edge information of the first VR scene using the Canny edge detection algorithm, and obtain multiple spatial regions based on the edge information;

[0090] The superpixel similarity between spatial regions is calculated, and the spatial regions with superpixel similarity greater than 0.7 are regarded as the same spatial region type. The calculation formula for superpixel similarity is:

[0091]

[0092] Among them, S i and Sj are the superpixel sets corresponding to the two spatial regions, W 1 is the diagonal weight matrix of color features, C i and C j S i and S j The corresponding color feature vector, σ 1 is the color similarity attenuation speed control coefficient, W 2 is the diagonal weight matrix of the position feature, L i and L j S i and S j The corresponding position eigenvector, σ 2 is the control coefficient of the position similarity decay speed, D ij For S i and S j The texture feature difference tensor, ‖D ij ‖ F D ij The Frobenius norm, σ 3 is the control coefficient of texture similarity decay speed.

[0093] In this embodiment, the method of filling VR elements for regions by using a variational autoencoder according to semantic information of different regions includes:

[0094] Analyze the characteristics of each spatial area, determine the semantic information, collect VR element sample data related to the semantic information, and perform preprocessing, including cropping, scaling, and normalization operations;

[0095] Construct a variational autoencoder model, including an encoder and a decoder, where the encoder extracts the features of the VR element sample data and maps them to the mean vector and variance vector to define the distribution of the latent space, expressed as:

[0096]

[0097] Among them, μ is the mean vector, σ 2 is the variance vector, Encoder is the encoder for encoding operation, X is the input data, L is the total number of encoder layers, w l and b l are the weight and bias of the lth layer, ξ is the ReLU activation function, BN means batch normalization operation, K l is the feature map of the lth layer, w k,l and b k,l are the weight and bias of the kth feature map in the lth layer, X k is the input of the kth feature map;

[0098] The variational autoencoder parameters are optimized according to the loss function, where the loss function consists of the reconstruction error and the KL divergence, and the expression is:

[0099]

[0100] Where N is the number of samples in the training data set, H, W and C are the height, width and number of channels of the image respectively, and x i,h,w,c is the original pixel value of the i-th sample image at the (h, w, c) position, is the pixel value of the i-th sample image at the (h, w, c) position generated by the decoder, z is the dimension of the latent space, μ j and σ j are the mean and variance of the jth dimension in the latent space, respectively. β is the balance coefficient, which is used to balance the relative importance of reconstruction loss and KL divergence in the total loss function and is set to 0.5.

[0101] After the decoder obtains the vector from the latent space sampling, it restores the VR elements of the original size. For dynamic VR elements, it adds dynamic effects on the basis of restoration based on the recorded motion trajectory information, and places the generated VR elements in the corresponding space area according to the actual layout and proportion.

[0102] In this embodiment, the method of mapping and associating the preset teaching knowledge graph with the second VR scene includes: analyzing the preset teaching knowledge graph structure to determine the knowledge node hierarchy and association, identifying the key elements corresponding to the knowledge graph nodes in the second VR scene, and combining the key elements in the second VR scene according to the logical relationship of the knowledge graph to form a teaching path.

[0103] In this embodiment, the method of constructing a recurrent neural network to learn the emotional state and perspective transfer state of the students on the teaching path includes:

[0104] Collect students' multimodal data on the teaching path, including behavioral data, facial data, and physiological data, input them into the pre-trained emotion classification model, output the probability distribution of the students' current emotional state, and select the emotional state with the highest probability as the emotional state prediction result;

[0105] A recurrent neural network model was constructed using a long short-term memory network. The multimodal data and the emotion state prediction results were combined as a training set to train the model. A softmax function was added to the output layer of the model to output the probability distribution of each viewing area. The expression of the probability distribution of the viewing area is:

[0106]

[0107] Among them, y t,jis the probability that the student's next perspective is in the jth region at time t, w j and w k are the weight vectors of the output layer corresponding to the j-th viewing area and the k-th viewing area, respectively. T is the transposition operation, o t,L is the output value of the last layer after the recurrent neural network is calculated at time t, c t,L is the cell state value of the last layer after the recurrent neural network calculation, b j and b k is the bias of the output layer corresponding to the j-th viewing area and the k-th viewing area, n is the total number of viewing areas divided in the VR scene, where o t,L and c t,L The calculation formula is:

[0108] o t，L =σ(W o，l ·[h t-1,l ;X t ]+b o，l )

[0109]

[0110] Among them, W o,l , W f,l , W i,l and Wc ,l are the weight matrices of the output gate, forget gate, input gate, and cell state update in layer l, respectively. o,l , b f,l , b i,l and b c,l and are the bias vectors of the output gate, forget gate, input gate and cell state update at layer l, respectively. R is the number of hidden layers. δ is the sigmoid activation function. h t-1,l is the hidden state of the lth layer at the previous moment t-1, X t is the eigenvector at time t, c t-1,L is the cell state value of the last layer of the recurrent neural network at time t-1.

[0111] In this embodiment, the method for updating the second VR scene based on the learned preference includes:

[0112] Judge the predicted emotional state output by the recurrent neural network. If the emotional state is positive, keep the current scene. If the emotional state is negative, adjust the scene color and layout.

[0113] According to the probability distribution of the viewing area output by the recurrent neural network model, the position of the VR element is adjusted. The expression is:

[0114]

[0115]

[0116] Among them, (x i ,y i ,z i ) is the original coordinate of the VR element, is the coordinates adjusted for VR elements, P j is the probability distribution value of the viewing area, θ h and θ v are the horizontal and vertical angles of the students’ perspective preferences, q is the number of VR elements, and e is the number of VR elements. i is the i-th VR element, A(e i ) is e i The distance from the current view center.

[0117] Figure 2 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 2 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0118] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0119] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0120] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a device for constructing an educational VR simulation scene based on artificial intelligence at the logical level. The processor executes the program stored in the memory and is specifically used to execute any of the aforementioned methods for constructing an educational VR simulation scene based on artificial intelligence.

[0121] The present invention can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0122] An embodiment of the present application also proposes a computer-readable storage medium, which stores one or more programs, and the one or more programs include instructions. When the instructions are executed by an electronic device including multiple applications, any one of the aforementioned methods for constructing an educational VR simulation scene based on artificial intelligence is executed.

[0123] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A method for constructing an educational VR simulation scene based on artificial intelligence, characterized in that: The following steps are involved: Acquire real image data, and fuse real image data of different resolutions and bands based on the pixel-level fusion method of convolutional neural network to obtain a multi-dimensional fused image; Constructing a GAN model including a generator network and a discriminator network, and using the generator network and the discriminator network to perform adversarial training based on the multivariate fusion image to obtain a first VR scene; Apply pixel segmentation to the first VR scene to identify different region types, and fill VR elements into the regions through a variational autoencoder according to semantic information of different regions to obtain a second VR scene; Mapping and associating the preset teaching knowledge graph with the second VR scene to obtain a teaching path; A recurrent neural network is constructed to learn the students' emotional state and perspective transfer state on the teaching path, and the second VR scene is updated based on the learned preference to obtain a final VR simulation scene.

2. The method for constructing an educational VR simulation scene based on artificial intelligence according to claim 1, characterized in that: The pixel-level fusion method based on convolutional neural network, a method for fusing real image data of different resolutions and different bands, includes: The U-Net architecture is selected to build a convolutional neural network, and images of different resolutions and bands are input into channels of different convolutional layers to extract image features, including texture features and shape features; Assign weights proportional to the resolution to the image features. The middle layer of the convolutional neural network performs weighted summation based on the weights of the image features to obtain a fused feature map. The fused feature map is upsampled and reconstructed to generate a multivariate fused image.

3. The method for constructing an educational VR simulation scene based on artificial intelligence according to claim 1, characterized in that: The method of using a generator network and a discriminator network to perform adversarial training based on the multivariate fusion image includes: The multi-dimensional fusion image is input into the generator network, and the image features are transformed and information is reorganized using a structure including a convolutional layer, a fully connected layer, and a ReLU activation function layer to generate an initial VR scene image. The initial VR scene image and the multi-element fusion image are input into the discriminator network, and features are extracted through the convolution layer and classified and judged through the fully connected layer, and the probability value of the output image is the real image; The loss function is constructed based on the probability value output by the discriminator network, and the expression is: Among them, m is the number of multivariate fusion images, z i is the i-th multivariate fusion image, G(z i ) is the generator network based on z i The generated image, D(G(z i )) is the discriminator network for G(z i ), n is the probability value of G(z i ), the number of features in x j is the jth feature point, is the discriminator network for G(z i ) in x j The gradient at The convergence condition is set to that the change in the loss function is less than 0.001 in 100 consecutive iterations, and iterative training is performed. In each iteration, the parameters in the generator network and the discriminator network are updated by back propagation.

4. The method for constructing an educational VR simulation scene based on artificial intelligence according to claim 1, characterized in that: The method of applying pixel segmentation to the first VR scene to identify different area types includes: Obtain edge information of the first VR scene using the Canny edge detection algorithm, and obtain multiple spatial regions based on the edge information; The superpixel similarity between spatial regions is calculated, and the spatial regions with superpixel similarity greater than 0.7 are regarded as the same spatial region type. The calculation formula for superpixel similarity is: Among them, S i and S j are the superpixel sets corresponding to the two spatial regions, W1 is the diagonal weight matrix of the color feature, C i and C j S i and S j The corresponding color feature vector, σ1 is the color similarity decay speed control coefficient, W2 is the diagonal weight matrix of the position feature, L i and L j S i and S j The corresponding position feature vector, σ2 is the control coefficient of the position similarity decay speed, D ij For S i and S j The texture feature difference tensor, ‖D ij ‖ F D ij , and σ3 is the control coefficient of the texture similarity decay rate.

5. The method for constructing an educational VR simulation scene based on artificial intelligence according to claim 1, characterized in that: The method of filling VR elements for regions by using a variational autoencoder according to semantic information of different regions includes: Analyze the characteristics of each spatial area, determine the semantic information, collect VR element sample data related to the semantic information, and perform preprocessing, including cropping, scaling, and normalization operations; Construct a variational autoencoder model, including an encoder and a decoder, where the encoder extracts the features of the VR element sample data and maps them to the mean vector and variance vector to define the distribution of the latent space, expressed as: Among them, μ is the mean vector, σ 2 is the variance vector, Encoder is the encoder for encoding, X is the input data, L is the total number of encoder layers, and w l and b l are the weight and bias of the lth layer, ξ is the ReLU activation function, BN means batch normalization operation, K l is the feature map of the lth layer, w k,l and b k,l are the weight and bias of the kth feature map in the lth layer, X k is the input of the kth feature map; The variational autoencoder parameters are optimized according to the loss function, where the loss function consists of the reconstruction error and the KL divergence, and the expression is: Where N is the number of samples in the training data set, H, W and C are the height, width and number of channels of the image respectively, and x i,h,w,c is the original pixel value of the i-th sample image at the (h, w, c) position, is the pixel value of the i-th sample image at the (h, w, c) position generated by the decoder, z is the dimension of the latent space, μ j and σ j are the mean and variance of the jth dimension in the latent space, respectively. β is the balance coefficient, which is used to balance the relative importance of reconstruction loss and KL divergence in the total loss function and is set to 0.

5. After the decoder obtains the vector from the latent space sampling, it restores the VR elements of the original size. For dynamic VR elements, it adds dynamic effects on the basis of restoration based on the recorded motion trajectory information, and places the generated VR elements in the corresponding space area according to the actual layout and proportion.

6. The method for constructing an educational VR simulation scene based on artificial intelligence according to claim 1, characterized in that: The method for mapping and associating the preset teaching knowledge graph with the second VR scene includes: analyzing the preset teaching knowledge graph structure to determine the knowledge node hierarchy and association, identifying the key elements corresponding to the knowledge graph nodes in the second VR scene, and combining the key elements in the second VR scene according to the logical relationship of the knowledge graph to form a teaching path.

7. The method for constructing an educational VR simulation scene based on artificial intelligence according to claim 1, characterized in that: The method of constructing a recurrent neural network to learn the emotional state and perspective transfer state of the students on the teaching path includes: Collect students' multimodal data on the teaching path, including behavioral data, facial data, and physiological data, input them into the pre-trained emotion classification model, output the probability distribution of the students' current emotional state, and select the emotional state with the highest probability as the emotional state prediction result; A recurrent neural network model was constructed using a long short-term memory network. The multimodal data and the emotion state prediction results were combined as a training set to train the model. A softmax function was added to the output layer of the model to output the probability distribution of each viewing area. The expression of the probability distribution of the viewing area is: Among them, y t,j is the probability that the student's next perspective is in the jth region at time t, w j and w k are the weight vectors of the output layer corresponding to the j-th viewing area and the k-th viewing area, respectively. T is the transposition operation, o t,L is the output value of the last layer after the recurrent neural network is calculated at time t, c t,L is the cell state value of the last layer after the recurrent neural network calculation, b j and b k is the bias of the output layer corresponding to the j-th viewing area and the k-th viewing area, n is the total number of viewing areas divided in the VR scene, where o t,L and c t,L The calculation formula is: the t,L =σ(W o,l ·[h t-1,l ;X t ]+b o,l ) Among them, W o,l , W f,l , W i,l and W c,l are the weight matrices of the output gate, forget gate, input gate, and cell state update in layer l, respectively. o,l , b f,l , b i,l and b c,l and are the bias vectors of the output gate, forget gate, input gate and cell state update at layer l, respectively. R is the number of hidden layers. δ is the sigmoid activation function. h t-1,l is the hidden state of the lth layer at the previous moment t-1, X t is the eigenvector at time t, c t-1,L is the cell state value of the last layer of the recurrent neural network at time t-1.

8. The method for constructing an educational VR simulation scene based on artificial intelligence according to claim 1, characterized in that: The method for updating the second VR scene based on the learned preference includes: Judge the predicted emotional state output by the recurrent neural network. If the emotional state is positive, keep the current scene. If the emotional state is negative, adjust the scene color and layout. According to the probability distribution of the viewing area output by the recurrent neural network model, the position of the VR element is adjusted. The expression is: Among them, (x i ,y i ,z i ) is the original coordinate of the VR element, is the coordinates adjusted for VR elements, P j is the probability distribution value of the viewing area, θ h and θ v are the horizontal and vertical angles of the students’ perspective preferences, q is the number of VR elements, and e is the number of VR elements. i is the i-th VR element, A(e i ) is e i The distance from the current view center.

9. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, enables the electronic device to execute the method according to any one of claims 1 to 8.

Images