Teaching event extraction method and device, equipment and storage medium

Abstract

The invention discloses a teaching event extraction method, device and equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the following steps: collecting a video sequence, an audio sequence and a text sequence in a teaching scene in real time, and inputting the video sequence, the audio sequence and the text sequence into a teaching event extraction model, a multi-level teaching event sequence is obtained; teaching events in the multi-level teaching event sequence are defined as nodes, semantic association of two adjacent teaching events is defined as edges, and a teaching event knowledge graph is constructed; and extracting a teaching event through the teaching event knowledge graph. According to the knowledge graph disclosed by the invention, the multi-dimension and time sequence of a plurality of teaching events and the complex relationship among the teaching events are considered, and the node information is updated according to the real-time teaching scene information, so that the teaching event extraction accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a teaching event extraction method, device, equipment and storage medium. Background Art

[0002] Generally, teaching events usually include teachers' explanations, students' questions, teacher-student interactions, students' feedback, and classroom discussions, which can reflect the dynamic changes in the teaching process. Teaching event extraction refers to the process of obtaining and identifying key teaching activities or events from the education process (such as classroom teaching and online learning). The accurate extraction of teaching events can not only help the evaluation of teaching quality, but also provide a basis for personalized teaching design. It is an important research topic in the fields of intelligent education, educational data analysis, and learning management systems.

[0003] In traditional teaching research and practice, teaching events mostly rely on the behavior of teachers and students, verbal communication, and changes in classroom atmosphere. However, with the development of educational informatization, especially the popularity of online learning platforms and smart classrooms, the complexity of teaching activities is also increasing. Therefore, how to effectively extract teaching events from multimodal information has become an important issue in research and application.

[0004] Knowledge graph is a technology that represents knowledge through graph structure, which is usually composed of entities, relationships and attributes, and can visualize and structure knowledge. By constructing a teaching knowledge graph, it is possible to systematically represent the concepts, events and interactions involved in the classroom, thereby supporting intelligent recommendations, personalized learning path planning, teaching evaluation and feedback, etc. Based on the extraction of teaching events, the construction of the knowledge graph uses natural language processing, information extraction and graph database technologies to extract entity relationships and form a complete teaching knowledge graph.

[0005] Although the existing knowledge graph-based teaching event extraction has made some progress, it still faces some shortcomings. In the process of knowledge graph construction, existing methods face great challenges in revealing the complex relationships between events. The associations between teaching events are often multidimensional and temporal, and traditional graph construction methods are usually unable to effectively model these deep relationships, resulting in a lack of in-depth understanding of teaching interactions. In addition, the constructed knowledge graph lacks a dynamic update mechanism and cannot be updated in real time with changes in teaching theories or the emergence of new educational research results, resulting in low accuracy of the teaching events finally extracted. Summary of the invention

[0006] Based on the above technical problems, the present invention provides a teaching event extraction method, device, equipment and storage medium, which solves the problem that the existing teaching event extraction methods are often unable to respond quickly to complex teaching situations, lack the ability to adapt to dynamic teaching situations, and cannot effectively model these deep-level relationships, resulting in a lack of in-depth understanding of teaching interactions, resulting in low accuracy of the teaching events finally extracted.

[0007] The present invention adopts the following technical solutions: In a first aspect, the present invention provides a teaching event extraction method, comprising the following steps: Real-time acquisition of video sequences, audio sequences and text sequences in teaching scenarios; Constructing a teaching event extraction model, wherein the teaching event extraction model includes a vector encoder, a variational quantum neural network, a multimodal fusion layer, and a quantum enhanced multi-level attention encoder; The video sequence, audio sequence and text sequence are input into the teaching event extraction model, and the video sequence, audio sequence and text sequence are vector-encoded by a vector encoder to obtain an audio vector, a video vector and a text vector; the audio vector, video vector and text vector are encoded, feature extracted and decoded by a variational quantum neural network to obtain an audio feature vector, a video feature vector and a text feature vector; the audio feature vector, video feature vector and text feature vector are fused by a multimodal fusion layer to obtain a fusion vector; the fusion vector and the marked teaching event sequence are feature extracted and fused by a quantum enhanced multi-level attention encoder to obtain a multi-level teaching event sequence; wherein the marked teaching event sequence is a plurality of teaching events sorted by time and labeled; The teaching events in the multi-level teaching event sequence are defined as nodes, and the semantic relationship between two adjacent teaching events is defined as edges to construct a teaching event knowledge graph; teaching events are extracted through the teaching event knowledge graph.

[0008] Preferably, the vector encoder includes an audio encoder, a video encoder and a text encoder, and the audio encoder, video encoder and text encoder are used to perform vector encoding on the video sequence, audio sequence and text sequence to obtain audio vectors, video vectors and text vectors.

[0009] Preferably, encoding, feature extraction and decoding the audio vector, the video vector and the text vector by a variational quantum neural network to obtain the audio feature vector, the video feature vector and the text feature vector comprises the following steps: The variational quantum neural network includes an encoding layer, an extraction layer and a decoding layer; Amplitude encoding is performed on the audio vector, the video vector and the text vector through the encoding layer to obtain a video quantum state, an audio quantum state and a text quantum state; The video quantum state, the audio quantum state and the text quantum state are subjected to feature extraction through the extraction layer to obtain video features, audio features and text features; The video features, audio features and text features are decoded through the decoding layer to obtain an audio feature vector, a video feature vector and a text feature vector.

[0010] Preferably, the label represents the category or feature of the teaching event, specifically including questioning, answering and discussion.

[0011] Preferably, the semantic relationship between any two adjacent events in the multi-level teaching event sequence is obtained through a spatiotemporal relationship structure encoder.

[0012] In a second aspect, the present invention provides a teaching event extraction device, comprising: The acquisition module is used to acquire video sequences, audio sequences and text sequences in the teaching scene in real time; A construction module is used to construct a teaching event extraction model, wherein the teaching event extraction model includes a vector encoder, a variational quantum neural network, a multimodal fusion layer, and a quantum enhanced multi-level attention encoder; An input module is used to input the video sequence, audio sequence and text sequence into the teaching event extraction model, and perform vector encoding on the video sequence, audio sequence and text sequence through a vector encoder to obtain an audio vector, a video vector and a text vector; encode, extract features and decode the audio vector, video vector and text vector through a variational quantum neural network to obtain an audio feature vector, a video feature vector and a text feature vector; fuse the audio feature vector, video feature vector and text feature vector through a multimodal fusion layer to obtain a fusion vector; perform feature extraction and fusion on the fusion vector and the marked teaching event sequence through a quantum enhanced multi-level attention encoder to obtain a multi-level teaching event sequence; wherein the marked teaching event sequence is a plurality of teaching events sorted by time and labeled; The extraction module is used to define the teaching events in the multi-level teaching event sequence as nodes, define the semantic relationship between two adjacent teaching events as edges, and construct a teaching event knowledge graph; and extract teaching events through the teaching event knowledge graph.

[0013] In a third aspect, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned teaching event extraction method when executing the program.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program implements the above-mentioned teaching event extraction method when executed by a processor.

[0015] Compared with the prior art, at least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects: The present invention takes into account that the association between teaching events has a temporal feature, and therefore firstly collects the video sequence, audio sequence and text sequence in the teaching scene in real time. Then the video sequence, audio sequence and text sequence are processed, and the complex relationship between each modal information is captured and fused to obtain a fusion vector. The fusion vector is subjected to feature extraction and fusion with the marked teaching event sequence to obtain a multi-level teaching event sequence. The present invention fuses real-time data with different teaching event templates to obtain a multi-level teaching event sequence that not only considers the time sequence of event occurrence, but also may include the complex relationship between events and context. Therefore, the present invention can extract teaching events in real time according to complex teaching situations, increase the adaptability to dynamic teaching situations, and the captured teaching events include multi-dimensional and temporal deep relationships.

[0016] Finally, the present invention defines the teaching events in the multi-level teaching event sequence as nodes, and defines the semantic association between two adjacent teaching events as edges, to construct a teaching event knowledge graph. The knowledge graph of the present invention takes into account the multi-dimensionality and temporal nature of multiple teaching events and the complex relationship between each teaching event, and updates the node information according to the real-time teaching scene information, thereby improving the accuracy of teaching event extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic diagram of a flow chart of a teaching event extraction method of the present invention; Figure 2 It is a flowchart of a teaching event extraction method of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Reference Figure 1 and Figure 2The present invention provides a teaching event extraction method, comprising the following steps: Step 1: Collect video sequences, audio sequences and text sequences in the teaching scene in real time.

[0021] The present invention efficiently extracts teaching event data from a teaching scene containing vision, audio and text. The association between teaching events is often temporal, that is, the video sequence, audio sequence and text sequence in the teaching scene are first collected in real time.

[0022] Step 2: Construct a teaching event extraction model, input the video sequence, audio sequence and text sequence into the teaching event extraction model, and obtain a multi-level teaching event sequence.

[0023] The present invention proposes a multimodal and multi-level teaching event extraction model, including a vector encoder, a variational quantum neural network, a multimodal fusion layer and a quantum enhanced multi-level attention (QMA) encoder, so as to achieve efficient extraction of teaching events. The model applies a variational quantum neural network to process the encoded quantum state data, by strategically inserting parameterized revolving gates around the entanglement gates, and iteratively layering these configurations to further capture complex data relationships. Then, the multi-level event sequence is obtained by using the multimodal embedding of the labeled teaching event sequence and the fusion through the QMA encoder. The present invention can parameterize the quantum circuit and adaptively adjust the revolving gate parameters to achieve deep fusion of complex data patterns, so as to achieve efficient extraction of teaching events.

[0024] Variational quantum neural network is a model that combines quantum computing and neural networks. It uses the characteristics of quantum computing to enhance the ability of machine learning, especially when dealing with complex data or optimization problems. The basic idea is to optimize the parameters of quantum circuits through variational methods so that the quantum state can better represent the given data distribution. In quantum computing, entanglement gates and rotation gates are two important quantum gates that are used to operate quantum bits and implement quantum algorithms. Entanglement gates are gates used to create entangled states between quantum bits. This gate can make the state of one quantum bit dependent on the state of another quantum bit. Rotation gates are quantum gates used to rotate a single quantum bit. Rotation gates can change the state of a quantum bit depending on its rotation angle and axis. Rotation gates can precisely control the state of a quantum bit by adjusting its parameters, allowing it to effectively transform in the quantum state space.

[0025] The QMA encoder, or quantum enhanced multi-level attention encoder, is an innovative technology that combines quantum computing with a multi-level attention mechanism. It enhances the feature extraction and fusion capabilities of multimodal data through the superposition and entanglement characteristics of quantum states.

[0026] The vector encoder includes an audio encoder, a video encoder and a text encoder. The audio encoder, the video encoder and the text encoder perform vector encoding on the video sequence, the audio sequence and the text sequence to obtain an audio vector, a video vector and a text vector.

[0027] The variational quantum neural network includes an encoding layer, an extraction layer, and a decoding layer. The encoding layer is used to perform amplitude encoding on the audio vector, video vector, and text vector to obtain the video quantum state, audio quantum state, and text quantum state. The extraction layer is a plurality of quantum circuits, which are used to input the video quantum state, audio quantum state, and text quantum state into the corresponding quantum circuits for feature extraction to obtain video features, audio features, and text features. The decoding layer is a measurement module, which is used to measure the state of the quantum circuit, that is, to decode the video features, audio features, and text features to obtain the audio feature vector, video feature vector, and text feature vector.

[0028] The audio feature vector, the video feature vector and the text feature vector are fused through a multimodal fusion layer to obtain a fusion vector.

[0029] The fused vector and the labeled teaching event sequence are input into the QMA encoder for feature extraction and fusion, and multiple multi-level teaching event sequences are obtained. A labeled teaching event sequence refers to a data structure that temporally sorts a series of events and attaches labels. These labels usually represent the category or characteristics of the event. For example, in a teaching scenario, they may include "question", "answer" and "discussion". Through labeling, these events become available for analysis and model training. A multi-level event sequence refers to an event sequence represented at multiple levels or dimensions, which not only considers the chronological order of events, but may also include complex relationships and contexts between events. For example, in the teaching process, a "question" event may have different levels of association with subsequent "answer" or "discussion" events. Through multi-level event sequences, the model can more comprehensively capture the semantics and structure of these events, thereby achieving a deeper understanding and analysis.

[0030] During the training process, we first need to extract vector features from multimodal data and encode them into quantum states. Next, we build a layered quantum circuit containing multiple parameterized revolving gates, each of which focuses on feature extraction at a different level. The output is calculated through forward propagation, and the loss is calculated. Then, the parameters of the revolving gate are optimized using backpropagation. A quantum-enhanced multi-level attention mechanism is introduced to effectively fuse the outputs of each layer. Finally, this process is iterated repeatedly until the model performance converges, thereby achieving deep capture of complex data relationships.

[0031] Step 3: Build a graph neural network model.

[0032] The complex structure and uncertain relationship of teaching events pose a challenge to identifying event relationships. This paper proposes a graph neural network model, including a spatiotemporal relationship structure encoder and a graph attention network. The spatiotemporal relationship structure encoder calculates the time ,space and a textual description of the teaching event To obtain fuzzy features (Multimodal membership function) and semantic relations, and the true degree of semantic relations is evaluated through fuzzy features: ; In the formula, is a fuzzy feature, For time, For space, is a text description of the teaching event, is the center point of each time feature, For the shape of each time feature, is the ambiguity parameter for each temporal feature, is the center point of each spatial feature, For the shape of each spatial feature, is the fuzziness parameter of each spatial feature, is the center point of each text feature, For the shape of each text feature, is the fuzziness parameter for each text feature. The semantic fuzzy graph attention network locates and aligns events based on the semantic relations (semantic embedding) and fuzzy features processed by the encoder. Positioning refers to locating the nodes that need to be acted on, that is, finding the specific target nodes. The alignment operation ensures that different types of relationships can be integrated in the graph to more fully express the correlation between nodes. The innovation of this method is that the encoder combines the fuzzy features of the event with the semantic relations to obtain more information and enhance the adaptability to complex relations.

[0033] The specific structure of the semantic fuzzy graph attention network includes an input layer, a graph convolution layer, and an output layer. First, the input layer receives the semantic relationship and fuzzy features from the encoder and represents each event as a node in the graph. Then, the graph structure is constructed, and the semantic relationship is defined as the edge between nodes to reflect the strength of the event relationship. Subsequently, the graph convolution layer updates the node representation by aggregating the information of neighboring nodes to enhance the ability to capture local structures. The graph attention mechanism is introduced to give adaptive weights to different edges and highlight important relationships. Finally, the fuzzy features are combined to handle ambiguity and evaluate uncertainty, and the output layer generates the final event location and alignment results. This design aims to improve the model's adaptability and understanding of complex relationships.

[0034] Step 4: Input the multi-level teaching event sequence into the graph neural network model.

[0035] The spatiotemporal structure encoder is used to extract semantic features (semantic relations) and fuzzy features of multiple multi-level teaching event sequences to obtain corresponding semantic features and fuzzy features. Multiple teaching events are defined as nodes through the input layer, and the semantic relations of multiple teaching events are defined as edges to obtain a graph structure. The semantic features corresponding to multiple nodes are aggregated through the graph convolution layer, and multiple nodes are updated. The real-time updated teaching event knowledge graph is output through the output layer, and finally the teaching events are extracted through the teaching event knowledge graph.

[0036] The above is a teaching event extraction method provided in one or more embodiments of this specification. Based on the same concept, the present invention also provides a teaching event extraction device, including a collection module, a construction module, an input module and an extraction module.

[0037] The acquisition module is used to acquire video sequences, audio sequences and text sequences in the teaching scene in real time.

[0038] The building module is used to construct a teaching event extraction model, which includes a vector encoder, a variational quantum neural network, a multimodal fusion layer, and a quantum enhanced multi-level attention encoder.

[0039] The input module is used to input the video sequence, audio sequence and text sequence into the teaching event extraction model, and perform vector encoding on the video sequence, audio sequence and text sequence through the vector encoder to obtain the audio vector, video vector and text vector; encode, feature extract and decode the audio vector, video vector and text vector through the variational quantum neural network to obtain the audio feature vector, video feature vector and text feature vector; fuse the audio feature vector, video feature vector and text feature vector through the multimodal fusion layer to obtain the fusion vector; perform feature extraction and fusion on the fusion vector and the marked teaching event sequence through the quantum enhanced multi-level attention encoder to obtain the multi-level teaching event sequence; wherein, the marked teaching event sequence is a plurality of teaching events sorted by time and labeled.

[0040] The extraction module is used to define the teaching events in the multi-level teaching event sequence as nodes, and the semantic relationship between two adjacent teaching events as edges, to construct a teaching event knowledge graph; and to extract teaching events through the teaching event knowledge graph.

[0041] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned teaching event extraction method when executing the program.

[0042] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned teaching event extraction method is implemented.

[0043] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0044] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A teaching event extraction method, characterized in that: The following steps are involved: Real-time acquisition of video sequences, audio sequences and text sequences in teaching scenarios; Constructing a teaching event extraction model, wherein the teaching event extraction model includes a vector encoder, a variational quantum neural network, a multimodal fusion layer, and a quantum enhanced multi-level attention encoder; The video sequence, the audio sequence and the text sequence are input into the teaching event extraction model, and the video sequence, the audio sequence and the text sequence are vector-encoded by a vector encoder to obtain an audio vector, a video vector and a text vector; The audio vector, video vector and text vector are encoded, feature extracted and decoded by a variational quantum neural network to obtain an audio feature vector, a video feature vector and a text feature vector; the audio feature vector, the video feature vector and the text feature vector are fused by a multimodal fusion layer to obtain a fusion vector; the fusion vector and the marked teaching event sequence are feature extracted and fused by a quantum enhanced multi-level attention encoder to obtain a multi-level teaching event sequence; wherein the marked teaching event sequence is a plurality of teaching events sorted by time and labeled; The teaching events in the multi-level teaching event sequence are defined as nodes, and the semantic relationship between two adjacent teaching events is defined as edges to construct a teaching event knowledge graph; teaching events are extracted through the teaching event knowledge graph.

2. A teaching event extraction method as claimed in claim 1, characterized in that: The vector encoder comprises an audio encoder, a video encoder and a text encoder, and the audio encoder, the video encoder and the text encoder are used to perform vector encoding on the video sequence, the audio sequence and the text sequence to obtain the audio vector, the video vector and the text vector.

3. A teaching event extraction method as claimed in claim 1, characterized in that: The method of encoding, feature extracting and decoding the audio vector, the video vector and the text vector by the variational quantum neural network to obtain the audio feature vector, the video feature vector and the text feature vector comprises the following steps: The variational quantum neural network includes an encoding layer, an extraction layer and a decoding layer; Amplitude encoding is performed on the audio vector, the video vector and the text vector through the encoding layer to obtain a video quantum state, an audio quantum state and a text quantum state; The video quantum state, the audio quantum state and the text quantum state are subjected to feature extraction through the extraction layer to obtain video features, audio features and text features; The video features, audio features and text features are decoded through the decoding layer to obtain an audio feature vector, a video feature vector and a text feature vector.

4. A teaching event extraction method as claimed in claim 1, characterized in that: The tags represent the categories or characteristics of teaching events, including questions, answers and discussions.

5. A teaching event extraction method as claimed in claim 1, characterized in that: The semantic relationship between any two adjacent events in a multi-level teaching event sequence is obtained through the spatiotemporal relationship structure encoder.

6. A teaching event extraction device, characterized in that: include: The acquisition module is used to acquire video sequences, audio sequences and text sequences in the teaching scene in real time; A construction module is used to construct a teaching event extraction model, wherein the teaching event extraction model includes a vector encoder, a variational quantum neural network, a multimodal fusion layer, and a quantum enhanced multi-level attention encoder; An input module is used to input the video sequence, audio sequence and text sequence into the teaching event extraction model, and to vector encode the video sequence, audio sequence and text sequence through a vector encoder to obtain an audio vector, a video vector and a text vector; The audio vector, video vector and text vector are encoded, feature extracted and decoded by a variational quantum neural network to obtain an audio feature vector, a video feature vector and a text feature vector; the audio feature vector, the video feature vector and the text feature vector are fused by a multimodal fusion layer to obtain a fusion vector; the fusion vector and the marked teaching event sequence are feature extracted and fused by a quantum enhanced multi-level attention encoder to obtain a multi-level teaching event sequence; wherein the marked teaching event sequence is a plurality of teaching events sorted by time and labeled; The extraction module is used to define the teaching events in the multi-level teaching event sequence as nodes, define the semantic relationship between two adjacent teaching events as edges, and construct a teaching event knowledge graph; and extract teaching events through the teaching event knowledge graph.

7. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for extracting teaching events described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the teaching event extraction method described in any one of claims 1 to 5 is implemented.

Images