Virtual teaching scene generation method and device, equipment and storage medium

Abstract

The invention discloses a virtual teaching scene generation method and device, equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the following steps: extracting semantic features, visual features and sampling features; mapping and classifying the plurality of features to obtain rendering parameters; the shapes and the surface textures of all the objects contained in the virtual classroom are rendered in real time through the rendering parameters, and three-dimensional teaching resources are obtained; spatial layout coding and scene hierarchical coding are respectively carried out on the three-dimensional teaching resources to obtain spatial layout features and hierarchical spatial features, the spatial layout features and the hierarchical spatial features are decoded to obtain geometric shapes and layout position parameters, and a virtual teaching scene is generated through the geometric shapes and the layout position parameters. Dynamic adjustment can be carried out according to different teaching activities and scenes, and the adaptability is improved. The mutual space relation of the multiple objects of the three-dimensional teaching resources is correctly described, and the accuracy of the generated teaching scene is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, equipment and storage medium for generating a virtual teaching scene. Background Art

[0002] With the rapid development of science and technology, the field of education is undergoing profound changes. As a new teaching environment, the virtual classroom is significantly different from the traditional classroom. It connects teachers and learners through digital platforms such as the Internet to achieve remote teaching and learning. Among them, the virtual teaching scene is the most important part of building a virtual classroom. Therefore, accurately generating a virtual teaching scene is of great significance to accurately building a virtual classroom.

[0003] The existing virtual teaching scene generation mainly adopts the 3D reconstruction method. 3D reconstruction is a multimedia technology widely used in computer-aided modeling and 3D animation. It can reconstruct 3D targets from single-view RGB images by using the assistance of shape prior and shape reasoning ability. In recent years, researchers have proposed a variety of improvement methods to improve the accuracy and efficiency of reconstruction. Han et al. explored the use of deep learning technology to estimate 3D shapes from single or multiple RGB images to solve some problems in this field. Sun et al. proposed an autoregressive point cloud generation model Point Grow, which models the three-dimensional spatial distribution in a point-by-point manner to achieve diversified and realistic 3D shape generation. Wu et al.'s research uses 2D diffusion prior to generate text to 3D content to improve the quality and details of the generated 3D model. A new method combining multiple noise estimation processes with pre-trained diffusion priors is proposed.

[0004] Although the virtual teaching scenes generated by the existing virtual teaching scene generation methods can accurately present the corresponding teaching scenes, these teaching scenes are mostly static information, with poor real-time and adaptability, and it is difficult to dynamically adjust according to different teaching activities and scenes. At the same time, when the above method is used to generate more complex teaching scenes, it is difficult to correctly describe the complicated mutual spatial relationships, which makes the final generated teaching scene deviate greatly from the actual one. Summary of the invention

[0005] The present invention provides a virtual teaching scene generation method, device, equipment and storage medium, which solves the problems that the teaching scenes generated by the existing virtual teaching scene generation methods are mostly static information, with poor real-time and adaptability, and when generating more complex teaching scenes, it is difficult to correctly describe the complicated mutual spatial relationships, so that the finally generated teaching scenes deviate greatly from the actual ones.

[0006] In a first aspect, the present invention provides a method for generating a virtual teaching scene, comprising the following steps: Extract features from text resources and picture resources containing teaching scenes to obtain semantic features, visual features and sampling features; the sampling features include the three-dimensional position and viewing direction of sampling points in the picture resources that change over time; Map and classify semantic features, visual features and sampling features to obtain rendering parameters of the virtual teaching scene; map the rendering parameters to three-dimensional space, and render the shapes and surface textures of all objects contained in the virtual classroom in real time through the rendering parameters to obtain three-dimensional teaching resources that change over time; The three-dimensional teaching resources are spatially encoded in layout and scene-level encoded to obtain spatial layout features for characterizing the layout position of the three-dimensional teaching resources in the three-dimensional scene and hierarchical spatial features for characterizing the dynamic evolution of the three-dimensional teaching resources in the three-dimensional scene. The spatial layout features and hierarchical spatial features are decoded to obtain geometric shapes and layout position parameters, and a virtual teaching scene is generated through the geometric shapes and layout position parameters.

[0007] Preferably, the feature extraction of the text resources and picture resources containing the teaching scenes to obtain semantic features, visual features and sampling features includes the following steps: Extract features from text resources through a text encoder to obtain semantic features; Extract features from image resources through a visual encoder to obtain visual features; The three-dimensional position and viewing direction of the sampling points in the image resources are obtained through the motion structure recovery method, and the time variable is added to obtain the sampling features.

[0008] Preferably, before performing spatial layout coding and scene level coding on the three-dimensional teaching resources, it is necessary to perform feature extraction on the three-dimensional teaching resources to obtain spatial point data, which includes the following steps: Select and enhance important feature dimensions in three-dimensional teaching resources through channel attention mechanism; Determine the key spatial areas in the spatial information of three-dimensional teaching resources through the spatial attention mechanism; Classify the three-dimensional teaching resources through the classifier and assign different types of resources to corresponding categories; The outputs of the channel attention mechanism, the spatial attention mechanism, and the classifier are fused and normalized to obtain spatial point data, which includes spatial position, category information, and key features.

[0009] Preferably, the three-dimensional teaching resources are respectively subjected to spatial layout coding and scene level coding, wherein the spatial layout coding of the three-dimensional teaching resources comprises the following steps: The spatial attention layer is used to capture the attention relationship between spatial points and obtain spatial features; Normalize the spatial features through the normalization layer; The normalized spatial features are given nonlinear features through the activation layer to obtain a spatial vector; The spatial vector is modeled by graph structure through hyperedge constraints, and the modeled data is input into the spatial attention layer, normalization layer and activation layer again to obtain the spatial layout features, which include the relative position, direction and size of the three-dimensional teaching resources.

[0010] Preferably, the three-dimensional teaching resources are respectively subjected to spatial layout coding and scene level coding, wherein the three-dimensional teaching resources are subjected to scene level coding, comprising the following steps: The initial feature processing module is used to extract features from the spatial points to obtain multiple scene features; Multiple scene features are fused through a feature fusion module to obtain fused features; The teacher's instructions are embedded through the embedding module to obtain word vectors; The fusion features and word vectors are used to capture scene information through long short-term memory networks to obtain scene temporal features. The scene temporal features are weighted by the attention mechanism to obtain hierarchical spatial features, which include the hierarchical order of the three-dimensional resources in time and the weight distribution of different teaching resources at different times.

[0011] Preferably, decoding the spatial layout features and the hierarchical spatial features to obtain geometric shape and layout position parameters comprises the following steps: The spatial layout features and hierarchical spatial features are linearly transformed and dimensionally adjusted through the input layer; By decoding the spatial layout features and hierarchical spatial features layer by layer, the geometric shape, size, direction and position parameters of the three-dimensional teaching resources are obtained; The spatial layout information and hierarchical spatial information are fused through the feature fusion layer to obtain unified spatial layout information.

[0012] Preferably, the rendering parameters include pixel value and volume density.

[0013] In a second aspect, the present invention provides a virtual teaching scene generation device, comprising: An extraction module is used to extract features from text resources and picture resources containing teaching scenes to obtain semantic features, visual features and sampling features; the sampling features include the three-dimensional position and viewing direction of the sampling points in the picture resources that change over time; A mapping module is used to map and classify semantic features, visual features and sampling features to obtain rendering parameters of the virtual teaching scene; the rendering parameters are mapped to three-dimensional space, and the shapes and surface textures of all objects contained in the virtual classroom are rendered in real time through the rendering parameters to obtain three-dimensional teaching resources that change over time; A generation module is used to perform spatial layout encoding and scene hierarchy encoding on three-dimensional teaching resources respectively, obtain spatial layout features for characterizing the layout position of three-dimensional teaching resources in three-dimensional scenes and hierarchical spatial features for characterizing the dynamic evolution of three-dimensional teaching resources in three-dimensional scenes, decode the spatial layout features and hierarchical spatial features, obtain geometric shapes and layout position parameters, and generate a virtual teaching scene through the geometric shapes and layout position parameters.

[0014] In a third aspect, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for generating a virtual teaching scene when executing the program.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned virtual teaching scene generation method is implemented.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention first extracts text resources and picture resources containing teaching scenes for feature extraction, and obtains semantic features, visual features and sampling features, wherein the sampling features include the three-dimensional position and viewing angle direction of the sampling points in the picture resources that change over time, so the features extracted by the present invention for constructing three-dimensional teaching resources are dynamic and real-time. These features are then mapped to obtain the rendering parameters of the virtual teaching scene, and the shapes and surface textures of all objects contained in the virtual classroom are rendered in real time through the rendering parameters to obtain three-dimensional teaching resources that change over time. The three-dimensional teaching resources of the present invention change in real time and can be dynamically adjusted according to different teaching activities and scenes, greatly improving adaptability.

[0017] At the same time, the spatial layout encoding and scene level encoding are performed on the three-dimensional teaching resources respectively, and the encoding results are decoded, and a virtual teaching scene is generated through the decoding results. The present invention can effectively utilize the spatial layout characteristics and hierarchical spatial characteristics of the three-dimensional teaching resources, can correctly describe the mutual spatial relationship of multiple objects in the three-dimensional teaching resources, and improve the accuracy of the generated teaching scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of a method for generating a virtual teaching scene according to the present invention; Figure 2 It is a schematic diagram of the structure of the multimodal neural radiation field model of the present invention; Figure 3 Perform feature extraction for the three-dimensional teaching resource of the present invention; Figure 4 It is a structural schematic diagram of the spatial layout encoder of the present invention; Figure 5 It is a schematic diagram of the structure of the scene level encoder of the present invention; Figure 6 This is a schematic diagram of a virtual teaching scene generated by the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Reference Figure 1 The present invention provides a method for generating a virtual teaching scene, which uses a multimodal neural radiation field model and dynamic space layout technology to transform text resources and image resources containing teaching scenes into an immersive and interactive three-dimensional virtual teaching environment to enhance learners' embodied cognitive experience. The method includes the following steps: S1: Extract features from text resources and image resources containing teaching scenes to obtain semantic features, visual features and sampling features.

[0022] Text and image resources containing teaching scenarios are obtained from existing knowledge graphs (Wikidata, CN-DBpedia), online education platforms (National Education Resources Public Service Platform, National Primary and Secondary School Smart Education Platform and First Courseware Network) or open education datasets (MOOCCube). There is a close relationship between text resources and image resources.

[0023] The theory of embodied cognition points out that learners’ understanding and memory of knowledge can be deepened through physical interactive experience. However, existing methods are out of touch with actual teaching practice when generating dynamic virtual teaching scenes, which affects learners’ embodied cognition level. Therefore, the present invention relies on educational concepts such as constructivism, integrates semantic features, visual features and sampling features of multi-perspective images, introduces the time dimension to expand the neural radiation field in space and time, and designs a multimodal neural radiation field model to synthesize dynamic three-dimensional virtual teaching resources that are both immersive and interactive.

[0024] Reference Figure 2 The multimodal neural radiation field model first converts text and pictures into semantic features containing educational information through the text encoder BERT and the visual encoder ViT respectively. and visual features . Where t is text and v is visual. Semantic features include semantic keywords containing educational information, vectors of characters and words, vector representation of content themes, part-of-speech tagging, semantic tagging and syntactic tagging, difficulty value or knowledge point type, etc. Visual features include the color, shape, texture and spatial position information of each object.

[0025] Then, the three-dimensional position (X) and viewing direction (d) of the sampling points in the image resources are analyzed by the motion structure recovery method, and the time variable (t) is added, thereby introducing the time dimension to expand the neural radiation field to the spatiotemporal level. By predicting the displacement of spatial points at different times: , enabling three-dimensional resources to change dynamically over time. This dynamic characteristic helps promote learners' internalization and long-term memory of knowledge.

[0026] In this embodiment, the sampling point refers to the three-dimensional spatial position in the image resource related to the virtual scene, which is used to describe the spatial information of the image resource. The sampling feature refers to the feature extracted from the sampling point information in the image resource, which contains the visual and spatial data related to the educational information in the scene. Specifically, the sampling features include the three-dimensional position (X), the viewing direction (d) and the time variable (t), which help to form the dynamic change information of each sampling point in the virtual teaching scene at different times. These sampling features are integrated into the neural radiation field to generate a three-dimensional teaching resource with spatiotemporal dynamics, thereby enhancing the learner's immersive experience and long-term memory of knowledge. Motion Structure Recovery is a computer vision technology used to recover the three-dimensional structure and camera motion information of the scene from an image or video sequence. This method is usually used for modeling dynamic scenes, especially when dealing with situations involving object motion, perspective changes, etc., and can effectively reconstruct the three-dimensional information of objects or scenes.

[0027] S2: Map and classify semantic features, visual features and sampling features to obtain rendering parameters of the virtual teaching scene.

[0028] After extracting sampling features , the semantic feature F t and visual features F v They are input as multimodal features to the fully connected layer, which maps and classifies the multimodal features and outputs pixel values ​​(c) and volume density (δ).

[0029] S3: Map the rendering parameters to the three-dimensional space, render the shapes and surface textures of all objects contained in the virtual classroom, and obtain three-dimensional teaching resources that change over time.

[0030] To solve the problem of low rendering efficiency of dynamic 3D resources, the pixel value and volume density are mapped into 3D space based on NeRF (Neural Radiance Fields) method. NeRF method can realize efficient generation of dynamic 3D teaching resources and support dynamic changes in time dimension. Learners can click on dynamic 3D resources to gain an in-depth understanding of the display and explanation of relevant knowledge points, thereby interacting with virtual scenes, which is in line with the concept of learners' autonomous construction of knowledge in constructivist theory.

[0031] Compared with traditional static 3D rendering methods, the NeRF method can take advantage of its volume rendering to maintain high-quality visual effects in dynamic scenes. For example, in a virtual teaching scene, objects (such as interactive teaching aids and teaching equipment) can be dynamically adjusted according to time, perspective changes, or learners' operations. This technology not only improves the real-time performance and visual immersion of rendering, but also enhances learners' knowledge internalization ability and learning effect through interactive learning methods.

[0032] In this embodiment, the three-dimensional resources of the equipment include some static objects in the classroom, such as desks, chairs, blackboards, desk and chair combinations, clocks and podiums. The three-dimensional resources of the course are dynamic teaching elements directly related to the course content. These resources are not limited to static objects, but are dynamically generated according to the different teaching contents. For example, earth models and microscope models. If the teaching content involves knowledge related to the earth or geography, the system will generate a three-dimensional earth model based on the course content for teachers to display, explain and interact in the virtual classroom. For chemistry courses, if experiments or explanations of the use of microscopes are required, the system will generate a three-dimensional microscope model based on the teaching content, and teachers can display and operate it as a teaching resource. Combine the podium and course resources to generate a teaching area. These dynamically generated three-dimensional resources can be flexibly provided according to the different needs of the course, helping teachers to conduct vivid teaching demonstrations and enhance students' interactivity and immersion.

[0033] In order to stimulate learners' enthusiasm for learning and cultivate their innovative thinking, this invention focuses on the layout of teaching resources in the virtual teaching space, aiming to create a diverse and content-rich virtual teaching environment. Therefore, a virtual teaching scene generation model is designed in combination with the teacher's instruction description, such as the teaching mode. By optimizing the layout of teaching resources in the virtual teaching space, a more immersive and spatially three-dimensional virtual teaching scene is generated.

[0034] S4: Extract features from three-dimensional teaching resources to generate a unified feature vector containing spatial position, category information and key features, providing input data for the subsequent spatial layout encoder.

[0035] Reference Figure 3 ,In the process of processing three-dimensional teaching resources, the channel attention mechanism, ,space attention mechanism and classifier extract and classify the ,features of three-dimensional teaching resources respectively. Then, through feature fusion and normalization ,operations, a unified feature vector is generated.

[0036] The channel attention mechanism selects and enhances important feature dimensions in the 3D resource data. Among the different feature channels of the 3D resource (such as color, texture, and size), it determines which channels are more important to the current task, thereby enhancing the representation ability of key features. The channel attention mechanism ensures that the model can focus on the most relevant features when processing resource data. The spatial attention mechanism determines the important spatial locations in the resource in the spatial information of the 3D resource. Through the spatial attention mechanism, the model can identify the key spatial areas in the 3D resource, such as the cushion part of the chair and the desktop area of ​​the table. This mechanism helps the model extract important information about each resource in space and provides support for subsequent layout. The classifier is used to classify the 3D resources and assign different types of resources to corresponding categories, such as classifying chairs, desks, and clocks into different types. By classifying the resources, the model can distinguish the functions and usage scenarios of different resources. For example, chairs and desks belong to the regular layout in the classroom, while the earth model or microscope belongs to the course resources. The output of the classifier is used to guide the subsequent spatial layout and scene construction. The features extracted by the channel and spatial attention mechanisms are fused to form a unified feature representation for each resource. After feature fusion, the scale of the features is adjusted through normalization operations (may use standardization or batch normalization) to ensure the stability of feature distribution, which is conducive to subsequent spatial layout prediction. The spatial point data output by this module contains the following key information of the three-dimensional resource: spatial location, category information (such as the functional type of the resource: chairs, desks, and experimental models, etc.) and key features (such as texture, material, and geometry, etc.). These spatial point data provide input for the spatial layout encoder to guide the reasonable layout of resources in the virtual scene.

[0037] S5: Input the spatial point data into the spatial layout encoding and scene hierarchy encoding respectively to obtain the spatial layout features and hierarchical spatial features, decode the spatial layout features and hierarchical spatial features to obtain geometric shapes and layout position parameters, and generate a virtual teaching scene through the geometric shapes and layout position parameters.

[0038] ,The scene construction model contains three core modules: spatial layout encoder, ,scene level encoder and layout level decoder. ,The spatial layout encoder takes the spatial point data in ,three-dimensional teaching resources as input and extracts the spatial layout features to ,predict the layout position of three-dimensional resources. ,In addition to considering the horizontal and vertical edge constraints of ,three-dimensional teaching resources, it is also necessary to realize the multi-relation encoding between ,resources according to the hyper-edge constraints between multiple resources to ensure that ,the spatial layout is more in line with the cognitive characteristics and ,laws of learners and reduce the cognitive load of learners.

[0039] Reference Figure 4 ,The spatial layout encoder includes a spatial attention layer, a normalization layer, an activation layer, and a hyperedge constraint.

[0040] The spatial attention layer is used to capture the attention relationship between spatial points to enhance the representation of important spatial features. Through the attention mechanism, the spatial locations with importance or priority in the scene are identified and focused on to ensure the rationality and coordination of resources during the layout process.

[0041] The standardization layer normalizes the spatial features to ensure that the distribution of features in the network is stable and reduce the risk of gradient disappearance or explosion. Through the standardization operation, the distribution of features between different layers is more stable, thereby improving the training effect and convergence speed of the model.

[0042] The activation layer uses the LeakyReLU activation function to give the network nonlinear characteristics, enabling it to better fit complex spatial layout relationships. LeakyReLU can handle negative values, thereby retaining the information of negative features and helping the model learn more complex spatial relationships.

[0043] After being processed by the attention layer, normalization layer, and activation layer, a spatial vector is generated to represent the spatial position and direction of the teaching resources.

[0044] Hyperedge constraints model space vectors through graph structures, combining the "hyperedge" constraint relationships between multiple resources. Hyperedge constraints refer to establishing multiple associations between different resources to ensure the rationality of resource layout and the consistency of mutual relationships.

[0045] After generating the preliminary spatial vector, the spatial features are further refined by passing through the spatial attention layer, normalization layer, and LeakyReLU activation. Through repeated attention mechanisms and feature extraction, the encoder is able to capture more detailed spatial relationships and generate more hierarchical spatial layout features.

[0046] After the above processing, the spatial layout features finally generated are used to indicate the specific layout position of the resources in the three-dimensional scene. These features can contain information such as the relative position, direction and size of the resources to ensure that the layout of the generated virtual teaching scene is reasonable.

[0047] Under the framework of embodied cognition theory, the teaching process is no longer just the transfer of knowledge, but gradually evolves into an integration of experience and perception over time. Therefore, the generation of dynamic three-dimensional scene sequences helps to guide learners to gradually deepen their learning cognition. The present invention is committed to generating a dynamic three-dimensional scene sequence model according to the teaching design in the virtual classroom, promoting the development of embodied cognition by creating highly interactive teaching tasks and activities, such as teaching simulation experiments, and ensuring that the presentation of different virtual scenes is consistent with the teaching process.

[0048] In the scene level encoder, the module encodes 3D resources at multiple levels through a variational autoencoder and recursively maps them into the scene space. Figure 5 ,The scene level encoder includes the initial feature processing module, feature fusion module, embedding module, long short-term memory network and attention mechanism.

[0049] The initial feature processing module includes convolution layer COV, residual connection Res, feature extraction layer CAP, full connection layer FC, ReLU and Sigmoid. Feature extraction is performed on the input data to obtain multiple scene features.

[0050] COV is used to extract the spatial features of the input data. Res is used to prevent the gradient vanishing problem and ensure that the features are not lost when passing through multiple layers. CAP is used to further extract high-level features. FC is used to convert the feature vector into an output of a specified dimension. ReLU is the ReLU activation function, which introduces nonlinear characteristics to the model. Sigmoid is the Sigmoid activation function, which is used to map the feature value to the range of (0,1).

[0051] The feature fusion module fuses the features extracted from different layers through addition and dot multiplication to generate a richer feature representation, namely the fused feature.

[0052] In the teacher instruction module, the teacher's instructions are embedded into word vectors as additional input information. Word vector embedding can transform the teacher's specific instructions (for example, teaching content or teaching objectives) into feature representations that the model can understand.

[0053] Multiple scene-level long short-term memory network (S-LSTM) modules are used to encode temporal features and obtain scene temporal features. S-LSTM can effectively capture dynamic changes and hierarchical structures in scenes. The design of multi-layer S-LSTM can help the model capture and store important scene information at different time steps.

[0054] The attention mechanism is used to focus on important locations or objects in the scene, and to weight the scene's temporal features to obtain hierarchical spatial features that characterize the dynamic evolution of 3D teaching resources in the 3D scene. The hierarchical spatial features include the hierarchical order of the 3D resources in time and the weight distribution of different teaching resources at different times. This ensures that the generated virtual teaching scene meets teaching needs. The attention mechanism can help the model focus on the key content or resources mentioned in the teacher's instructions and give priority to these elements when generating the layout.

[0055] Finally, the layout level decoder decodes the spatial layout and scene level encodings top-down into fine-grained geometry and layout position parameters.

[0056] The layout hierarchical decoder includes input layer, layer-by-layer decoding and feature fusion layer.

[0057] Input layer: The spatial layout features from the spatial layout encoder and the hierarchical spatial features from the scene level encoder are input through the input layer. These input data will undergo a simple linear transformation and dimension adjustment before entering the decoder to ensure that the features can be correctly processed during the decoding process. The purpose of this step is to standardize the input features to make them suitable for subsequent decoding and fusion processes.

[0058] Layer-by-layer decoding: In the layer-by-layer decoding layer, the layout hierarchy decoder gradually decodes the input features according to the hierarchical structure to obtain high-level global layout information and fine-grained local geometric parameters. The decoder first generates the geometric shape (such as cylinder, rectangle, etc.), size, direction and three-dimensional position parameters of each teaching resource. This process ensures that the layout of each resource is hierarchical and consistent, and gradually refines to local spatial details based on global information to ensure that the resources in the virtual scene are accurately and reasonably distributed.

[0059] Feature fusion layer: After decoding layer by layer, the feature fusion layer fuses the features from the spatial layout encoder and the scene hierarchy encoder to generate unified spatial layout information. Through feature fusion, the decoder combines the spatial layout information and hierarchical spatial information to ensure that the distribution and relative position relationship of resources in space remain consistent and meet the teaching objectives. The fused features will further adjust the spatial position of the resources and dynamically adjust the layout of the resources based on the timing information to ensure that the layout and order of the resources in the virtual teaching scene can change reasonably over time, thereby enhancing the interactivity and hierarchy of the teaching process.

[0060] Application to scene generation: These decoding results will be used for the final rendering of the virtual teaching scene, such as Figure 6 As shown, it ensures that each teaching resource is accurately placed in the virtual environment to enhance the teaching effect and learners' interactive experience.

[0061] Compared with the existing virtual scene generation methods, the virtual teaching scene layout model proposed in the present invention can effectively utilize the spatial position hierarchical relationship of three-dimensional teaching resources and the multi-contextual relationship between resources, solve the problem of the disconnection between spatial layout and actual teaching practice, generate diversified virtual teaching scenes with reasonable layout, and promote learners' embodied cognitive development and immersive learning.

[0062] Based on the same concept, the present invention also provides a virtual teaching scene generation device, including an extraction module, a mapping module and a generation module.

[0063] The extraction module is used to extract features from text resources and image resources containing teaching scenes to obtain semantic features, visual features and sampling features; the sampling features include the three-dimensional position and viewing direction of the sampling points in the image resources that change over time.

[0064] The mapping module is used to map and classify semantic features, visual features and sampling features to obtain rendering parameters of the virtual teaching scene; the rendering parameters are mapped to three-dimensional space, and the shapes and surface textures of all objects contained in the virtual classroom are rendered in real time through the rendering parameters to obtain three-dimensional teaching resources that change over time.

[0065] The generation module is used to perform spatial layout encoding and scene hierarchy encoding on the three-dimensional teaching resources respectively, to obtain spatial layout features for characterizing the layout position of the three-dimensional teaching resources in the three-dimensional scene and hierarchical spatial features for characterizing the dynamic evolution of the three-dimensional teaching resources in the three-dimensional scene, to decode the spatial layout features and hierarchical spatial features, to obtain geometric shapes and layout position parameters, and to generate a virtual teaching scene through the geometric shapes and layout position parameters.

[0066] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned virtual teaching scene generation method when executing the program.

[0067] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned virtual teaching scene generation method is implemented.

[0068] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0069] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for generating a virtual teaching scene, characterized in that: The following steps are involved: Extract features from text resources and picture resources containing teaching scenes to obtain semantic features, visual features and sampling features; the sampling features include the three-dimensional position and viewing direction of sampling points in the picture resources that change over time; Map and classify semantic features, visual features and sampling features to obtain rendering parameters of the virtual teaching scene; Mapping rendering parameters to three-dimensional space, using rendering parameters to render the shapes and surface textures of all objects in the virtual classroom in real time, and obtaining three-dimensional teaching resources that change over time; The three-dimensional teaching resources are spatially encoded in layout and scene-level encoded to obtain spatial layout features for characterizing the layout position of the three-dimensional teaching resources in the three-dimensional scene and hierarchical spatial features for characterizing the dynamic evolution of the three-dimensional teaching resources in the three-dimensional scene. The spatial layout features and hierarchical spatial features are decoded to obtain geometric shapes and layout position parameters, and a virtual teaching scene is generated through the geometric shapes and layout position parameters.

2. A method for generating a virtual teaching scene according to claim 1, characterized in that: The feature extraction of the text resources and the picture resources containing the teaching scenes to obtain semantic features, visual features and sampling features includes the following steps: Extract features from text resources through a text encoder to obtain semantic features; Extract features from image resources through a visual encoder to obtain visual features; The three-dimensional position and viewing direction of the sampling points in the image resources are obtained through the motion structure recovery method, and the time variable is added to obtain the sampling features.

3. A method for generating a virtual teaching scene according to claim 1, characterized in that: Before performing spatial layout coding and scene level coding on the three-dimensional teaching resources, it is necessary to extract features of the three-dimensional teaching resources to obtain spatial point data, including the following steps: Select and enhance important feature dimensions in three-dimensional teaching resources through channel attention mechanism; Determine the key spatial areas in the spatial information of three-dimensional teaching resources through the spatial attention mechanism; Classify the three-dimensional teaching resources through the classifier and assign different types of resources to corresponding categories; The outputs of the channel attention mechanism, the spatial attention mechanism, and the classifier are fused and normalized to obtain spatial point data, which includes spatial position, category information, and key features.

4. A method for generating a virtual teaching scene as claimed in claim 3, characterized in that: The three-dimensional teaching resources are respectively subjected to spatial layout coding and scene level coding, wherein the three-dimensional teaching resources are subjected to spatial layout coding, including the following steps: The spatial attention layer is used to capture the attention relationship between spatial points and obtain spatial features; Normalize the spatial features through the normalization layer; The normalized spatial features are given nonlinear features through the activation layer to obtain a spatial vector; The spatial vector is modeled by graph structure through hyperedge constraints, and the modeled data is input into the spatial attention layer, normalization layer and activation layer again to obtain the spatial layout features, which include the relative position, direction and size of the three-dimensional teaching resources.

5. A method for generating a virtual teaching scene as claimed in claim 3, characterized in that: The three-dimensional teaching resources are respectively subjected to spatial layout coding and scene level coding, wherein the three-dimensional teaching resources are subjected to scene level coding, including the following steps: The initial feature processing module is used to extract features from the spatial points to obtain multiple scene features; Multiple scene features are fused through a feature fusion module to obtain fused features; The teacher's instructions are embedded through the embedding module to obtain word vectors; The fusion features and word vectors are used to capture scene information through long short-term memory networks to obtain scene temporal features. The scene temporal features are weighted by the attention mechanism to obtain hierarchical spatial features, which include the hierarchical order of the three-dimensional resources in time and the weight distribution of different teaching resources at different times.

6. A method for generating a virtual teaching scene as claimed in claim 5, characterized in that: Decoding the spatial layout features and the hierarchical spatial features to obtain geometric shapes and layout position parameters includes the following steps: The spatial layout features and hierarchical spatial features are linearly transformed and dimensionally adjusted through the input layer; By decoding the spatial layout features and hierarchical spatial features layer by layer, the geometric shape, size, direction and position parameters of the three-dimensional teaching resources are obtained; The spatial layout information and hierarchical spatial information are fused through the feature fusion layer to obtain unified spatial layout information.

7. A method for generating a virtual teaching scene according to claim 1, characterized in that: The rendering parameters include pixel value and volume density.

8. A virtual teaching scene generation device, characterized in that: include: An extraction module is used to extract features from text resources and picture resources containing teaching scenes to obtain semantic features, visual features and sampling features; the sampling features include the three-dimensional position and viewing direction of the sampling points in the picture resources that change over time; A mapping module is used to map and classify semantic features, visual features and sampling features to obtain rendering parameters of the virtual teaching scene; the rendering parameters are mapped to three-dimensional space, and the shapes and surface textures of all objects contained in the virtual classroom are rendered in real time through the rendering parameters to obtain three-dimensional teaching resources that change over time; A generation module is used to perform spatial layout encoding and scene hierarchy encoding on three-dimensional teaching resources respectively, obtain spatial layout features for characterizing the layout position of three-dimensional teaching resources in three-dimensional scenes and hierarchical spatial features for characterizing the dynamic evolution of three-dimensional teaching resources in three-dimensional scenes, decode the spatial layout features and hierarchical spatial features, obtain geometric shapes and layout position parameters, and generate a virtual teaching scene through the geometric shapes and layout position parameters.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for generating a virtual teaching scene as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method for generating a virtual teaching scene according to any one of claims 1 to 7 is implemented.

Images