Virtual teaching scene description generation method and device, equipment and storage medium

Abstract

The invention discloses a virtual teaching scene description generation method, device and equipment and a storage medium, and relates to the technical field of knowledge maps, and the method comprises the following steps: respectively obtaining comparison cross parameters of corresponding modes of a time sequence vector, a text vector, a picture vector and an audio vector, and constructing a cross attention matrix based on a plurality of comparison cross parameters; normalizing the plurality of modal vectors through a cross attention matrix to obtain a plurality of normalized modal vectors; iterating a cross attention matrix construction process and a normalization process on the plurality of normalized modal vectors to obtain a plurality of optimal modal vectors, and fusing and mapping the optimal modal vectors to obtain a multi-modal feature vector; the multi-modal feature information is decoded, virtual teaching scene description is generated, and the virtual teaching scene description comprises text information and picture information. According to the invention, deep and accurate fusion of subject knowledge information and teaching events is realized, and finally, high-quality, fine-grained, reasonable and complete virtual teaching scene description is generated.

Description

Technical Field

[0001] The present invention relates to the field of knowledge graph technology, and in particular to a method, device, equipment and storage medium for generating a virtual teaching scene description. Background Art

[0002] As a new type of teaching environment, the virtual classroom is significantly different from the traditional classroom. It connects teachers and learners through digital platforms such as the Internet to achieve remote teaching and learning. The virtual classroom has the characteristics of cross-time and space, flexibility and personalization. It can be carried out in various forms such as video conferencing, online classrooms and virtual laboratories to meet the needs of different disciplines and teaching contents. From the perspective of educational theory, the virtual classroom breaks through the time and space limitations of traditional education, leads the transformation of the educational paradigm, provides a more open learning environment, and promotes the sharing and exchange of knowledge. In practical applications, the virtual classroom makes up for the problem of uneven regional educational resources and has important social significance and practical value. In the process of building a virtual classroom, obtaining a scene description of virtual teaching is a very critical step.

[0003] Teaching activities guided by teaching events are the key to ensuring clear and realistic scene design and visual experience of virtual teaching scenes. At the same time, the generation of virtual teaching scenes also relies on multimodal subject knowledge information to generate accurate and reasonable virtual teaching scene descriptions, thereby constructing vivid and realistic teaching scenes. How to effectively integrate multimodal subject knowledge information and teaching events is a key technical issue in generating high-quality rich text virtual teaching scene descriptions.

[0004] Although the virtual teaching scene description generated by existing technologies (such as aggregation and alignment-based strategies) can fuse multimodal information, these fusion processes often only stay at the surface level, lack in-depth analysis of the intrinsic relationship between data, and cannot be accurately fused. At the same time, only the complete process of teaching events is considered, ignoring its temporal nature, which leads to the problem of data information loss during the fusion process, making the generated virtual teaching scene description inconsistent with the actual teaching situation. Summary of the invention

[0005] The present invention provides a method, device, equipment and storage medium for generating a virtual teaching scene description, which solves the problem that the prior art lacks in-depth analysis of the intrinsic relationship between data in generating a virtual teaching scene description and that data information is easily lost during the fusion process, resulting in the generated virtual teaching scene description being inconsistent with the actual teaching situation.

[0006] The present invention adopts the following technical solutions: In a first aspect, the present invention provides a method for generating a virtual teaching scene description, comprising the following steps: Extract the time series information of teaching events to obtain the time series vector of teaching events; encode the subject knowledge information to obtain the text vector, picture vector and audio vector of the subject knowledge information; Respectively obtain the contrast cross parameters of the corresponding modes of the time series vector, text vector, picture vector and audio vector, and construct a cross attention matrix based on multiple contrast cross parameters; respectively normalize the time series vector, text vector, picture vector and audio vector through the cross attention matrix to obtain multiple normalized modal vectors; iterate the above cross attention matrix construction and normalization process for multiple normalized modal vectors to obtain multiple optimal modal vectors, fuse and map the multiple optimal modal vectors to obtain a multimodal feature vector; wherein the cross attention matrix is ​​used to characterize the correlation strength of different modal information features; The multimodal feature vectors are weighted to obtain the feature vectors of the multimodal data; the feature vectors of the multimodal data are subjected to intra-modal and inter-modal feature interactions to obtain multimodal feature information; The multimodal feature information is decoded to generate a virtual teaching scene description, wherein the virtual teaching scene description includes text information and picture information.

[0007] Preferably, the step of extracting the timing information of the teaching events to obtain the timing vectors of the teaching events comprises the following steps: Timestamp coding is used to assign time tags to the boundaries of teaching events to obtain time series data; the time tags include the start time and end time of the teaching event; The time series data is decomposed into multiple components of different frequencies through wavelet transform, and the multiple components are filtered to obtain the time series vector of teaching events.

[0008] Preferably, the cross attention matrix is ​​constructed based on multiple comparison cross parameters, as shown below: ; In the formula, is the cross attention matrix, d is the feature dimension, is the text comparison cross parameter, is the cross-parameter for image comparison, is the audio comparison cross parameter, is the cross-parameter for time series comparison, T is transposed.

[0009] Preferably, the multiple modal vectors are normalized respectively by a cross attention matrix, as shown below: ; In the formula, is the normalized representation, is the normalization function, For modal x The vector representation of {text q ,picture i ,Audio v ,Time Series s}.

[0010] Preferably, performing intra-modal and inter-modal feature interactions on feature vectors of multimodal data to obtain multimodal feature information comprises the following steps: Decomposing the feature vector of the multimodal data into two iterative feature vectors, and fusing the two iterative feature vectors to obtain a fused feature vector; Inputting two iterative feature vectors, a fused feature vector, and external information into an N-layer network, wherein the external information includes semantic information, contextual background, or word embedding vectors of multimodal data; Obtain an iterative feature vector of a next layer based on an iterative feature vector of a current layer and a corresponding external input, and obtain a fused feature vector of a next layer based on the iterative feature vector of the next layer and a fused feature vector of the current layer; The above iteration is performed on the fused feature vector and iterative feature vector of the next layer to obtain multimodal feature information.

[0011] Preferably, the intra-modal and inter-modal features are interacted in an N-layer network through deep recursive operations, and the iterative feature vector of the next layer is obtained based on the iterative feature vector of the current layer and the corresponding external input, as shown below: ; In the formula, For the j Layer Mode x The iterated eigenvector of For external information, For the j Layer Mode x Iterated eigenvectors of ; The method of obtaining the fused feature vector of the next layer based on the iterative feature vector of the next layer and the fused feature vector of the current layer is specifically as follows: ; In the formula, For modal x The feature fusion weights, is the weight parameter, For the j Comprehensive feature representation of the layer.

[0012] Preferably, the teaching events include teaching preparation, learning tasks and teacher-student activities; the subject knowledge information includes knowledge concepts, course content, subject hierarchy, teaching resources and subject attributes.

[0013] In a second aspect, the present invention provides a virtual teaching scene description generating device, comprising: The encoding module is used to extract the time sequence information of the teaching events to obtain the time sequence vector of the teaching events; encode the subject knowledge information to obtain the text vector, picture vector and audio vector of the subject knowledge information; A construction module is used to respectively obtain the contrast cross parameters of the corresponding modes of the time series vector, text vector, picture vector and audio vector, and construct a cross attention matrix based on multiple contrast cross parameters; normalize the time series vector, text vector, picture vector and audio vector respectively through the cross attention matrix to obtain multiple normalized modal vectors; iterate the above cross attention matrix construction and normalization process for multiple normalized modal vectors to obtain multiple optimal modal vectors, fuse and map the multiple optimal modal vectors to obtain a multimodal feature vector; wherein the cross attention matrix is ​​used to characterize the correlation strength of different modal information features; An interaction module is used to weight the multimodal feature vectors to obtain the feature vectors of the multimodal data; perform intra-modal and inter-modal feature interactions on the feature vectors of the multimodal data to obtain multimodal feature information; The decoding module is used to decode the multimodal feature information to generate a virtual teaching scene description, wherein the virtual teaching scene description includes text information and picture information.

[0014] In a third aspect, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for generating a description of a virtual teaching scene when executing the program.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method for generating a description of a virtual teaching scene is implemented.

[0016] Compared with the prior art, at least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects: The present invention aims at the problem that the prior art does not consider the timing of teaching events. First, the timing information of teaching events is extracted to obtain the timing vector of teaching events. The contrast cross parameters of the corresponding modes of the text vector, picture vector and audio vector of the teaching event timing vector and the subject knowledge information are obtained. Then, a cross-attention matrix is ​​constructed based on multiple contrast cross parameters. Through the cross-attention matrix, each modal information can be weighed and compared, and the internal relationship between the data can be deeply analyzed to generate a more comprehensive and richer multimodal feature vector. The timing vector, text vector, picture vector and audio vector are normalized respectively through the cross-attention matrix to obtain multiple normalized modal vectors. Iteration of the above process highlights the correlation relationship between each modality, thereby avoiding the loss of modal data. Finally, multiple optimal modal vectors are fused and mapped to obtain a multimodal feature vector.

[0017] The multimodal feature vectors are weighted to obtain the feature vectors of the multimodal data; the feature vectors of the multimodal data are subjected to intra-modal and inter-modal feature interactions to achieve a deep and precise fusion of subject knowledge information and teaching events, and ultimately generate a high-quality, fine-grained and reasonably complete description of the virtual teaching scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of a method for generating a virtual teaching scene description of the present invention; Figure 2 It is a process schematic diagram of the multi-modal temporal interactive feature fusion method of the present invention; Figure 3 It is a process diagram of the enhanced deep attention balance method of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] In order to enable those skilled in the art to fully understand how to implement the present invention, this section provides an explanation of the embodiments of the claims.

[0022] Figure 1 It is a process schematic diagram of the method for integrating multimodal subject knowledge information and teaching events of the present invention; Figure 2 It is a process schematic diagram of the multi-modal temporal interactive feature fusion method of the present invention; Figure 3 The following is a schematic diagram of the process of strengthening the deep attention balance method of the present invention. Figure 1-Figure 3 A method for generating a virtual teaching scene description provided by an embodiment of the present invention is described in detail, and specifically comprises the following steps: S1: Extract the time series information of teaching events to obtain the time series vector of teaching events; encode the subject knowledge information to obtain the text vector, picture vector and audio vector of the subject knowledge information.

[0023] In order to solve the problem of reduced quality of scene generation description caused by ignoring the correlation between the two, refer to Figure 2 The present invention designs a method for integrating multimodal subject knowledge information and teaching events. The method aims to maintain the data information characteristics of subject knowledge information and teaching events, and utilizes the knowledge graph structure of the knowledge base to enhance the intrinsic connection between them, so as to promote the generation of rich text virtual teaching scene descriptions (including teaching objectives, learner analysis, teaching content organization, teaching methods and scene details description, etc.) in the form of pictures and texts, thereby providing data support for the construction of virtual classroom scenes.

[0024] Teaching events include time-sequential activities such as teaching preparation, learning tasks, and teacher-student activities, while existing multimodal subject knowledge bases generally associate teaching resources such as knowledge concepts and course content in the form of knowledge graphs. Both are very important for shaping realistic and vivid descriptions of teaching scenarios. Therefore, teaching events are identified and extracted from a variety of teaching resources such as teaching cases, lesson records, and micro-classes. At the same time, a new multimodal subject knowledge base that conforms to the laws of teaching is constructed based on the attributes of the subject (such as content, scope, structure, and development laws) and teaching objectives and other knowledge information. It will also draw on some existing open knowledge graphs (construction methods) to enrich the content and structure of the knowledge base and provide data support for the subsequent generation of virtual teaching scene descriptions.

[0025] Multimodal subject knowledge information specifically refers to the core content related to a specific subject, including knowledge concepts, course content, teaching rules, subject hierarchy, teaching resources and subject attributes. This information covers important knowledge points, chapter structure and teaching objectives in the syllabus, and is presented in a structured manner through knowledge graphs to support the generation of virtual classrooms. For example, resources such as textbooks, courseware and videos in the course can be used as basic data for virtual teaching scenes to ensure the logic and hierarchy of the content, so that the teaching scenes are highly consistent with the teaching objectives and learners' needs, and generate accurate and dynamic virtual classroom experiences.

[0026] The subject knowledge information in the multimodal subject knowledge base is encoded to obtain text vectors, picture vectors and audio vectors.

[0027] Traditional multimodal data fusion cannot simultaneously consider the multimodal information of subject knowledge information and the temporal information of teaching events, which leads to the problem of data information loss during the fusion process. Therefore, the present invention designs a multimodal temporal interactive feature fusion method, which compares the information within and between each modality and fuses to obtain a more comprehensive and richer data feature representation.

[0028] like Figure 2 As shown in the figure, before fusion, it is necessary to accurately capture the time series information in the teaching events, so a time series dynamic mapping method is proposed. This method uses timestamp coding to assign time tags to the boundaries of teaching events, obtains time series data, clearly identifies the start and end of the event, and provides an accurate basis for time series analysis. Subsequently, the time series data is decomposed into components of different frequencies through wavelet transform and the time series information is enhanced through filtering, so as to fully mine the time series information of the teaching events to generate time series vectors. Timestamp coding is a technology that assigns time tags to events or data. By associating a unique time point with each event, the occurrence time, start and end time of the event are clearly identified. In the timestamp coding process of teaching events, the specific operation is to assign a time tag to each teaching event (such as teacher explanation and teacher-student interaction) to define the time range or sequence of the event.

[0029] After the teaching events are time-stamped, the time tag of each event is obtained. This information forms an accurate time series data (S 1 S 2 S 3 …S n ), which provides a reliable basis for subsequent timing analysis and modeling. Through these time tags, the order and duration of events can be established, and the scene presentation in the virtual classroom can be ensured to be consistent with the actual teaching steps.

[0030] In order to fully tap into these time series information, the present invention adopts the method of wavelet transform. Wavelet transform can decompose time series data into components of different frequencies, helping to capture the subtle time changes and frequency characteristics in teaching events. Then, these components are processed by filtering technology to enhance the clarity of time series information. These steps ultimately generate a time series vector, which can represent the dynamic characteristics of teaching events and ensure the continuity and rationality of the teaching process when the virtual classroom scene is generated.

[0031] S2: Obtain the contrast cross parameters of the corresponding modes of the time series vector, text vector, picture vector and audio vector respectively, and construct a cross-attention matrix based on multiple contrast cross parameters; normalize, fuse and map the time series vector, text vector, picture vector and audio vector respectively through the cross-attention matrix to obtain a multimodal feature vector.

[0032] In order to deal with the problem of modal data loss in the fusion process, the present invention designs a modal collaborative enhancement network, which includes N iterative modules (modal information comparison layer, cross attention module and normalization layer). At the same time, the temporal order is taken into account as an additional modal information input network, and the network obtains the contrast cross parameters of the corresponding modalities through the modal information comparison layer. {text q ,picture i ,Audio v ,Time Series s}, and then use the cross-attention module to build the cross-attention matrix A mut , calculated as follows: ; In the formula, is the cross-attention matrix, which is used to describe the relationship strength of different modal information features; d is the feature dimension, is the text comparison cross parameter, is the cross-parameter for image comparison, is the audio comparison cross parameter, is the cross-parameter for time series comparison, T is transposed.

[0033] Then through Each modality data is normalized to achieve cross-modality calibration and ensure data consistency and comparability. Represents the vector representation of the corresponding modality. Iterating the above process highlights the correlation between each modality, thereby avoiding the loss of modal data. Finally, the data of each modality is fused through the mapping layer to obtain a multimodal feature vector. Compared with existing methods, thanks to the knowledge graph structure of the knowledge base itself, the multimodal time series interaction feature fusion method can extract time series information through the time series dynamic mapping method and the modal collaborative enhancement network, while weighing and comparing the complementary information of each modality, thereby generating a more comprehensive and richer feature representation.

[0034] S3: Weight the multimodal feature vectors to obtain the weighted representation of the multimodal data feature vectors; interact the features within and between modalities in a multi-layer network through deep recursive operations to obtain multimodal feature information.

[0035] In order to ensure the generation of reasonable high-quality rich text descriptions of virtual teaching scenarios, on the basis of obtaining a relatively rich and comprehensive information feature representation, it is also necessary to enhance the structural correlation between subject knowledge information and events to solve the problem of data feature fuzzification caused by redundant information. Figure 3 As shown in the figure, the present invention designs a method to strengthen the deep attention balance to emphasize the hierarchical nature of knowledge understanding and application, and realizes the deep integration of multimodal subject knowledge information and teaching events through feature selection. Obtain the weighted representation of the new multimodal data feature vector by weighting In order to achieve deep fusion of multimodal data features, deep recursive operations are used to interact intra-modal and inter-modal features in a multi-layer network. The specific process includes: Decompose the feature vector of multimodal data into two iterative feature vectors ( and ), and fuse the two iterative feature vectors to obtain the fused feature vector ( ).

[0036] The two iterative feature vectors, the fused feature vector and the external information ( X 1 and X 2 ) is input to the N-layer network.

[0037] Get the iterative feature vector of the next layer based on the iterative feature vector of the current layer and the corresponding external input ( and ), based on the iterative feature vector of the next layer and the fused feature vector of the current layer, the fused feature vector of the next layer is obtained ( ).

[0038] The above iteration is performed on the fused feature vector and iterative feature vector of the next layer to obtain multimodal feature information.

[0039] In the interaction process of each layer of the network, a dynamic feature selection mechanism is introduced to calculate the importance of modal features, thereby determining the contribution between modalities. The calculation process of the dynamic feature selection mechanism is as follows: ; ; ; In the formula, Indicates j Layer Mode x In the recursive process, the feature vector of each layer will be updated; For modal x The eigenvector of Transform and update. The function is to transform the features of the current layer and external input Input together to generate new feature representation; Represents external information input during the recursive process, which may be additional information related to the current modality or features of other modalities. This information plays an auxiliary role in the recursive update process; Representing modality x The feature fusion weight of x The proportion in the fusion process determines the influence of different modalities; is the fusion function, which is used to integrate the fusion features of the current layer With modal x In the j +1 layer feature vector Combine to generate new feature weight representation; Indicated in j The layer obtains a comprehensive feature representation by fusing information from different modalities. It is an intermediate result in the recursive process and represents the integration of multimodal information. is the weight parameter in the fusion, which is used to adjust the proportion of different modes in the fusion process. The weights between modes can be balanced to avoid information loss; j Comprehensive characteristics of the layer and the updated modal x feature Add element by element to ensure that feature information is not lost in multi-layer recursion.

[0040] In this embodiment, the additional information related to the current modality is supplementary data directly related to the features of a certain modality (such as text, picture, audio or time series). For example, when processing the text modality, additional semantic information, contextual background or word embedding vectors related to the text may be considered. The purpose of this additional information is to enhance the expressive power of the current modality during the fusion process and avoid information loss. The features of other modalities refer to the fact that when the current modality is performing feature fusion, in addition to using the data of its own modality, it also refers to the data of other modalities. For example: when processing the text modality, the visual features of the picture modality may be combined to supplement the context. When processing the audio modality, the semantic features of the text modality may be combined to improve the accuracy of the expression.

[0041] When the feature weights reach a balanced state hour, is the multimodal feature vector fused in the previous layer, which is the feature representation obtained through the interaction and fusion of modal information in the previous layer of the network. is the input feature of the current layer, that is, the initial feature directly output from the modal information comparison layer without recursive fusion. In the hybrid layer, the element-level weight fusion technology is designed, that is, the residual operation of element-by-element multiplication and summation avoids the gradient vanishing and gradient exploding problems, and the result is passed to the next layer. By repeating the above operations, the correlation information within and between modalities can be obtained from each level to achieve deep fusion of multimodal information.

[0042] S4: Decode the multimodal feature information to generate a virtual teaching scene description, which includes text information and picture information.

[0043] The multimodal feature information is processed in a refined manner with the help of a large language model, and the processed information is passed to the image decoder and text decoder to generate a rich text information description that is highly consistent with the teaching scene (teaching objectives, learner analysis, teaching content organization, scene detail description, and teaching methods). Compared with existing methods, the enhanced deep attention balance method uses a dynamic feature selection mechanism and element-level weight fusion technology to deeply understand and capture the correlation between multimodal data to balance the information contribution between each modality, thereby improving the fusion effect, and using a large language model and decoder to generate a rich text virtual teaching scene description containing text and multi-view images.

[0044] Based on the same concept, the present invention also provides a virtual teaching scene description generation device, including an encoding module, a construction module, an interaction module and a decoding module.

[0045] The encoding module is used to extract the time series information of teaching events to obtain the time series vectors of teaching events; and to encode the subject knowledge information to obtain the text vectors, picture vectors and audio vectors of the subject knowledge information.

[0046] The construction module is used to obtain the comparative cross-parameters of the corresponding modes of the time series vector, text vector, picture vector and audio vector respectively, and construct a cross-attention matrix based on multiple comparative cross-parameters; the time series vector, text vector, picture vector and audio vector are normalized respectively through the cross-attention matrix to obtain multiple normalized modal vectors; the multiple normalized modal vectors are iterated through the above cross-attention matrix construction and normalization process to obtain multiple optimal modal vectors, and the multiple optimal modal vectors are fused and mapped to obtain a multimodal feature vector; wherein the cross-attention matrix is ​​used to characterize the correlation strength of different modal information features.

[0047] The interaction module is used to weight the multimodal feature vectors to obtain the feature vectors of the multimodal data; and to perform intra-modal and inter-modal feature interactions on the feature vectors of the multimodal data to obtain multimodal feature information.

[0048] The decoding module is used to decode the multimodal feature information and generate a virtual teaching scene description, which includes text information and picture information.

[0049] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for generating a description of a virtual teaching scene when executing the program.

[0050] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned virtual teaching scene description generation method is implemented.

[0051] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0052] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for generating a description of a virtual teaching scene, characterized in that: The following steps are involved: Extract the time series information of teaching events to obtain the time series vector of teaching events; encode the subject knowledge information to obtain the text vector, picture vector and audio vector of the subject knowledge information; Obtain comparative cross-parameters of the corresponding modes of the time series vector, text vector, picture vector and audio vector respectively, and construct a cross-attention matrix based on multiple comparative cross-parameters; normalize the time series vector, text vector, picture vector and audio vector respectively through the cross-attention matrix to obtain multiple normalized modal vectors; Iterate the above cross-attention matrix construction and normalization process for multiple normalized modal vectors to obtain multiple optimal modal vectors, and fuse and map the multiple optimal modal vectors to obtain a multimodal feature vector; wherein the cross-attention matrix is ​​used to characterize the correlation strength of different modal information features; The multimodal feature vectors are weighted to obtain the feature vectors of the multimodal data; the feature vectors of the multimodal data are subjected to intra-modal and inter-modal feature interactions to obtain multimodal feature information; The multimodal feature information is decoded to generate a virtual teaching scene description, wherein the virtual teaching scene description includes text information and picture information.

2. A method for generating a virtual teaching scene description as claimed in claim 1, characterized in that: The step of extracting the time sequence information of the teaching events to obtain the time sequence vector of the teaching events comprises the following steps: Timestamp coding is used to assign time tags to the boundaries of teaching events to obtain time series data; the time tags include the start time and end time of the teaching event; The time series data is decomposed into multiple components of different frequencies through wavelet transform, and the multiple components are filtered to obtain the time series vector of teaching events.

3. A method for generating a virtual teaching scene description as claimed in claim 1, characterized in that: The cross-attention matrix is ​​constructed based on multiple comparison cross parameters, as shown below: ； In the formula, is the cross attention matrix, d is the feature dimension, is the cross-parameter for text comparison, is the cross-parameter for image comparison, is the audio comparison cross parameter, is the cross-parameter for time series comparison, T is transposed.

4. A method for generating a virtual teaching scene description as claimed in claim 3, characterized in that: The multiple modal vectors are normalized respectively by the cross attention matrix, as shown below: ； In the formula, is the normalized representation, is the normalization function, For modal x The vector representation of {text q ,picture i ,Audio v ,Time Series s }.

5. A method for generating a virtual teaching scene description as claimed in claim 4, characterized in that: The step of performing intra-modal and inter-modal feature interactions on the feature vectors of the multimodal data to obtain multimodal feature information includes the following steps: Decomposing the feature vector of the multimodal data into two iterative feature vectors, and fusing the two iterative feature vectors to obtain a fused feature vector; Inputting two iterative feature vectors, a fused feature vector, and external information into an N-layer network, wherein the external information includes semantic information, contextual background, or word embedding vectors of multimodal data; Obtain an iterative feature vector of a next layer based on an iterative feature vector of a current layer and a corresponding external input, and obtain a fused feature vector of a next layer based on the iterative feature vector of the next layer and a fused feature vector of the current layer; The above iteration is performed on the fused feature vector and iterative feature vector of the next layer to obtain multimodal feature information.

6. A method for generating a virtual teaching scene description as claimed in claim 5, characterized in that: The interaction of intra-modal and inter-modal features in the N-layer network is performed through deep recursive operations, and the iterative feature vector of the next layer is obtained based on the iterative feature vector of the current layer and the corresponding external input, as shown below: ； In the formula, For the j Layer Mode x The iterated eigenvector of For external information, For the j Layer Mode x Iterated eigenvectors of ; The method of obtaining the fused feature vector of the next layer based on the iterative feature vector of the next layer and the fused feature vector of the current layer is specifically as follows: ； In the formula, For modal x The feature fusion weights are is the weight parameter, For the j Comprehensive feature representation of the layer.

7. A method for generating a virtual teaching scene description as claimed in claim 1, characterized in that: The teaching events include teaching preparation, learning tasks and teacher-student activities; the subject knowledge information includes knowledge concepts, course content, subject hierarchy, teaching resources and subject attributes.

8. A virtual teaching scene description generating device, characterized in that: include: The encoding module is used to extract the time sequence information of the teaching events and obtain the time sequence vector of the teaching events; Encode the subject knowledge information to obtain the text vector, picture vector and audio vector of the subject knowledge information; A construction module is used to obtain the contrast cross parameters of the corresponding modes of the time series vector, text vector, picture vector and audio vector respectively, and construct a cross attention matrix based on multiple contrast cross parameters; the time series vector, text vector, picture vector and audio vector are normalized respectively through the cross attention matrix to obtain multiple normalized modal vectors; Iterate the above cross-attention matrix construction and normalization process for multiple normalized modal vectors to obtain multiple optimal modal vectors, and fuse and map the multiple optimal modal vectors to obtain a multimodal feature vector; wherein the cross-attention matrix is ​​used to characterize the correlation strength of different modal information features; An interaction module is used to weight the multimodal feature vectors to obtain the feature vectors of the multimodal data; perform intra-modal and inter-modal feature interactions on the feature vectors of the multimodal data to obtain multimodal feature information; The decoding module is used to decode the multimodal feature information to generate a virtual teaching scene description, wherein the virtual teaching scene description includes text information and picture information.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for generating a description of a virtual teaching scene as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating a virtual teaching scene description as described in any one of claims 1 to 7 is implemented.

Images