Dynamic classroom optimization method based on real-time space-time semantic graph tracking

Abstract

The invention discloses a dynamic classroom optimization method based on real-time space-time semantic graph tracking, and relates to the field of artificial intelligence, and the method comprises the steps: obtaining first multi-modal data of a current classroom in a classroom teaching process in response to a teaching optimization request; mapping each piece of first multi-modal data into a node in a first space-time semantic graph, generating the first space-time semantic graph based on a preset edge construction condition and a preset edge generation condition, and then inputting the first space-time semantic graph into a preset graph neural network to obtain a first dynamic score vector in the classroom teaching process; and then, based on a preset strategy network, according to the first dynamic score vector and the target score corresponding to the first dynamic score vector, a classroom improvement strategy is generated, real-time sensing and dynamic scoring of the classroom teaching process are realized, a more accurate classroom improvement strategy is obtained, and the classroom teaching efficiency is improved. Therefore, on the basis of improving the real-time performance and comprehensiveness of course teaching evaluation, the teaching scheme can be adjusted in real time, and the classroom teaching quality is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking. Background Art

[0002] Classroom teaching evaluation plays a crucial role in classroom teaching, helping teachers better understand students' learning progress, optimize educational programs, and help students achieve better learning outcomes. In recent years, classroom teaching evaluation has gradually evolved from traditional manual observation and questionnaires to automated and intelligent methods. These methods utilize devices such as cameras, microphones, and interactive terminals to collect audio, video, and behavioral data from the classroom, enabling the perception and analysis of the teaching process.

[0003] Current classroom teaching evaluation technologies, on the one hand, are mostly "post-event" evaluations, failing to provide real-time feedback during the teaching process. On the other hand, crude data fusion strategies often overlook the deeper relationships between temporal, spatial, and semantic meanings, failing to fully perceive the classroom teaching process. Based on existing classroom teaching evaluations, it is difficult to adjust teaching plans and optimize teaching content in real time based on classroom teaching evaluations, which affects the quality of classroom teaching.

[0004] Therefore, relevant technologies still need to be improved and developed. Summary of the Invention

[0005] The main purpose of this application is to provide a dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking, aiming to solve the technical problem in related technologies that it is difficult to adjust teaching plans and optimize teaching content in real time based on classroom teaching evaluation.

[0006] In order to achieve the above objectives, the first aspect of the present application provides a dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking, wherein the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking includes: In response to the teaching optimization request, obtaining first multimodal data of the current class during the classroom teaching process; Mapping each piece of the first multimodal data into a node in a first spatiotemporal semantic graph, and generating the first spatiotemporal semantic graph based on a preset edge construction condition and a preset edge generation condition; Inputting the first spatiotemporal semantic graph into a preset graph neural network to obtain a first dynamic scoring vector during classroom teaching; Based on a preset strategy network and according to the first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector, a classroom improvement strategy is generated.

[0007] Optionally, in response to the teaching optimization request, obtaining first multimodal data of the current class during the classroom teaching process includes: In response to the teaching optimization request, starting a preset cache detection module; Through the cache detection module, query whether there is cache data of the teaching optimization request; If the cached data of the teaching optimization request does not exist, the multimodal data acquisition module is started to obtain the first multimodal data of the current class during the classroom teaching process through the multimodal data acquisition module.

[0008] Optionally, the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking further includes: If the cached data of the teaching optimization request exists, the pre-improvement strategy in the cached data is set as the class improvement strategy of the current class and announced.

[0009] Optionally, the first multimodal data includes: at least one of text data, a sentiment score vector, and a video feature vector, as well as an event timestamp and spatial location information in the current classroom; The acquiring, by the multimodal data acquisition module, the first multimodal data of the current class during the classroom teaching process includes: The multimodal data acquisition module is used to collect classroom activity data of the current class, wherein the classroom activity data includes: voice data, video data, interaction logs, and spatial location information of the current class; wherein the spatial location information is obtained according to the image acquisition device in the multimodal acquisition module; Converting the voice data into text to obtain the text data; Obtaining the emotion score vector according to the acoustic parameters of the text data and the speech data using a preset text emotion model; and Extracting facial expression features and scene features of each frame of the video data, and generating a video feature vector based on the facial expression features and scene features; and The interaction log is analyzed to determine valid events in each time period, and the time to which the valid event belongs is set as the event timestamp.

[0010] Optionally, mapping each piece of the first multimodal data into a node in a first spatiotemporal semantic graph, and generating the first spatiotemporal semantic graph based on a preset edge construction condition and a preset edge generation condition, includes: Mapping each piece of the first multimodal data into a node of a first spatiotemporal semantic graph; According to the preset edge construction conditions, respectively calculating the time difference, spatial Euclidean distance, and semantic similarity of the two nodes; According to the time difference, the spatial Euclidean distance, and the semantic similarity, constructing a directed edge of the first spatiotemporal semantic graph from two of the nodes that meet at least two of the preset edge generation conditions to obtain the first spatiotemporal semantic graph; The preset edge generation conditions include: the time difference is less than or equal to a preset time difference threshold, the spatial Euclidean distance is less than or equal to a preset distance threshold, and the semantic similarity is greater than or equal to a preset similarity.

[0011] Optionally, after constructing a directed edge of the first spatiotemporal semantic graph from two nodes that meet at least two preset edge generation conditions according to the time difference, the spatial Euclidean distance, and the semantic similarity to obtain the first spatiotemporal semantic graph, the method further includes: Determining a teaching scenario in the classroom teaching process, wherein the teaching scenario includes at least one of lectures, question-answering, and discussions; The first spatiotemporal semantic graph is clustered according to the teaching scene to obtain a semantic subgraph of each teaching scene, and each semantic subgraph is set as a first spatiotemporal semantic graph.

[0012] Optionally, generating a classroom improvement strategy based on a preset strategy network and according to the first dynamic scoring vector and a target score corresponding to the first dynamic scoring vector includes: Obtaining the policy network corresponding to the first spatiotemporal semantic graph; and Calculating a deviation between the first dynamic scoring vector and a target score corresponding to the first dynamic scoring vector, and setting the deviation as the current class state; The current classroom state is input into the strategy network corresponding to the first spatiotemporal semantic graph to generate the classroom improvement strategy.

[0013] Optionally, after generating the classroom improvement strategy, the method further includes: In response to an improvement instruction based on the classroom improvement strategy feedback, obtaining second multimodal data of the classroom teaching process within a preset time period; wherein the improvement instruction is used to indicate that the teacher is detected to be implementing the classroom improvement strategy; Mapping each piece of the second multimodal data into a node in a second spatiotemporal semantic graph, and generating the second spatiotemporal semantic graph based on a preset edge construction condition and a preset edge generation condition; Inputting the second spatiotemporal semantic graph into the graph neural network to generate a second dynamic scoring vector; The policy network is updated according to the second dynamic scoring vector and a target score corresponding to the second dynamic scoring vector.

[0014] In the second aspect, the present application provides a terminal device, wherein the smart terminal includes a memory, a processor, and a computer program stored on the memory and runnable on the processor, and when the computer program is executed by the processor, it implements any step of the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking.

[0015] The third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements any one of the steps of the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking.

[0016] In order to solve the above problems, in the present application scheme, in response to a teaching optimization request, the first multimodal data of the current class in the classroom teaching process is obtained; each first multimodal data is mapped to a node in the first spatiotemporal semantic graph, and the first spatiotemporal semantic graph is generated based on preset edge construction conditions and preset edge generation conditions, and then the first spatiotemporal semantic graph is input into a preset graph neural network to obtain the first dynamic scoring vector in the classroom teaching process; then based on the preset strategy network, and according to the first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector, a classroom improvement strategy is generated.

[0017] Compared with the existing technology, in the solution corresponding to the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking provided by this application, a first spatiotemporal semantic graph reflecting the teaching process is constructed through the first multimodal data in the classroom teaching process. In this technology, the first spatiotemporal semantic graph is subjected to data fusion and feature extraction through the graph neural network to obtain the first dynamic scoring vector in the classroom teaching process. Then, through the preset strategy network, according to the first dynamic scoring vector and the corresponding target score, the corresponding classroom improvement strategy is generated to guide the adjustment of classroom teaching content. Through the above solution, real-time perception and dynamic scoring of the classroom teaching process are realized, and more accurate classroom improvement strategies are obtained, so as to improve the real-time, comprehensive and guiding nature of the course teaching evaluation and adjust the teaching plan in real time, optimize the teaching content and improve the quality of classroom teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 This is a flow chart of a dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking provided by an embodiment of the present application; Figure 2 This is another flow chart of a dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking provided by an embodiment of the present application; Figure 3 This is a block diagram of the internal structure principle of a smart terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it should be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0021] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0022] It should also be understood that the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0024] As used in this specification and the appended claims, the term "if" can be interpreted as meaning "when" or "upon" or "in response to determining" or "in response to being classified into," depending on the context. Similarly, the phrase "if it is determined" or "if it is classified into [described condition or event]" can be interpreted as meaning "upon determination" or "in response to determining" or "upon classification into [described condition or event]" or "in response to being classified into [described condition or event]," depending on the context.

[0025] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0027] With advances in science and technology, classroom teaching evaluation is gradually evolving from traditional methods like manual observation and questionnaires to automated and intelligent approaches. Current research is utilizing devices such as cameras, microphones, and interactive terminals to capture classroom audio, video, and behavioral data to enable perception and analysis of the teaching process. For example, an intelligent teaching system integrating multimodal behavioral perception technology proposes using deep learning models to analyze indicators such as student-student interaction, facial expressions, and speech frequency. A classroom teaching evaluation model is constructed by integrating sentiment analysis and speech recognition to enhance the ability to capture student engagement. While these methods have improved the objectivity and automation of classroom evaluation to a certain extent, they still have significant shortcomings. For one thing, existing technologies are mostly "post-event" evaluations, failing to provide real-time feedback during the classroom. Furthermore, data fusion strategies are crude and often overlook deep temporal, spatial, and semantic relationships, making it difficult to fully capture classroom interaction.

[0028] Based on the existing classroom teaching evaluation, it is difficult to adjust the teaching plan and optimize the teaching content in real time according to the classroom teaching evaluation, which affects the quality of classroom teaching.

[0029] In order to solve the above problems, in the present application scheme, in response to a teaching optimization request, the first multimodal data of the current class in the classroom teaching process is obtained; each first multimodal data is mapped to a node in the first spatiotemporal semantic graph, and the first spatiotemporal semantic graph is generated based on preset edge construction conditions and preset edge generation conditions, and then the first spatiotemporal semantic graph is input into a preset graph neural network to obtain the first dynamic scoring vector in the classroom teaching process; then based on the preset strategy network, and according to the first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector, a classroom improvement strategy is generated.

[0030] Compared with the existing technology, in the solution corresponding to the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking provided by this application, a first spatiotemporal semantic graph reflecting the teaching process is constructed through the first multimodal data in the classroom teaching process. In this technology, the first spatiotemporal semantic graph is subjected to data fusion and feature extraction through a graph neural network to obtain a first dynamic scoring vector in the classroom teaching process. Then, through a preset strategy network, a corresponding classroom improvement strategy is generated according to the first dynamic scoring vector and the corresponding target score to guide the adjustment of classroom teaching content. Through the above solution, real-time perception and dynamic scoring of the classroom teaching process are realized, and more accurate classroom improvement strategies are obtained, so that on the basis of improving the real-time, comprehensiveness and guidance of course teaching evaluation, the teaching plan can be adjusted in real time, the teaching content can be optimized, and the quality of classroom teaching can be improved.

[0031] This application provides a dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking, such as Figure 1 As shown, the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking provided by this application includes the following steps: S101, in response to a teaching optimization request, obtaining first multimodal data of a current class during a classroom teaching process.

[0032] Specifically, the above-mentioned step S101 can be implemented in the following manner: in response to the teaching optimization request, start the preset cache detection module; through the cache detection module, query whether there is cache data of the teaching optimization request; if there is no cache data of the teaching optimization request, start the multimodal data acquisition module to obtain the first multimodal data of the current class during the classroom teaching process through the multimodal data acquisition module; if there is cache data of the teaching optimization request, set the preset improvement strategy in the cache data as the classroom improvement strategy of the current class and announce it.

[0033] In an embodiment of the present application, different business scenarios are pre-set, and the above business scenarios at least include: a fixed feedback scenario, a local feedback scenario or a cloud scenario.

[0034] Fixed feedback scenario: refers to the automatic triggering of teaching optimization requests at predefined fixed time points (such as every 10 minutes after the start of the course), without the need for administrators or teachers to actively initiate; Local feedback scenario: This refers to a teaching optimization request initiated by a teacher or administrator through a terminal or mobile device in the classroom, with the local server completing data collection and optimization strategy broadcasting. Cloud scenario: This refers to a scenario where data is uploaded from the classroom to the cloud platform for processing, the cloud server generates classroom improvement strategies, and sends the results to the local terminal. It is usually used in remote classrooms or resource-constrained environments.

[0035] It can be understood that, from the different business scenarios in the embodiments of the present application, it can be learned that the above-mentioned teaching optimization request can be automatically triggered or passively triggered.

[0036] In an embodiment of the present application, when a processing device (e.g., a local server, a cloud server, an edge computing box, etc.) receives a teaching optimization request, it first activates a preset cache detection module in response to the teaching optimization request. The cache detection module queries whether cached data generated by the teaching optimization request exists, where the cached data includes at least a dynamic evaluation vector or a classroom improvement strategy. If the cache detection module detects cached data for the teaching optimization request, it directly calls the classroom improvement strategy in the cached data to publish it, thereby reducing the computational burden and improving the response speed.

[0037] The teaching optimization request may include at least one query identifier among the teacher ID, course ID, and class ID. It is understood that the teaching optimization request in the embodiment of the present application may be composed of the teacher ID, course ID, and class ID together, that is, the teaching optimization request is a query variable formed by the concatenation of the teacher ID, course ID, and class ID.

[0038] Furthermore, in an embodiment of the present application, the course-optimized classroom teaching data is stored as cached data, and a preset expiration time is set. The cached data may include key information such as the teacher ID, course ID, class ID, classroom improvement strategy, and time window (voice fingerprint may also be added). After receiving a teaching optimization request, the cached data corresponding to the teaching optimization request is queried based on the query identifier in the teaching optimization request. If the query is successful, a determination is made as to whether the verification time to live (TTL) value exceeds the preset expiration time. If the TTL exceeds the preset expiration time, it is determined that the cached data for the teaching optimization request does not exist. If the TTL does not exceed the preset expiration time, it is determined that the cached data for the teaching optimization request exists.

[0039] For example, if a teacher has previously requested a classroom improvement strategy for this course, the system first constructs a unique cache key based on key information such as the teacher ID, course ID, class ID, and time window (which can also include a voice fingerprint). The cache layer then queries the entry, verifying whether the Time To Live (TTL) value has expired (or whether the optional audio fingerprint is consistent). If the query is successful and the verification passes, the cache is determined to be "valid," meaning that cached data exists for the teaching optimization request. In this case, the preset improvement strategy in the cached data is directly used as the classroom improvement strategy for the teaching optimization request, without the need for re-collection and analysis.

[0040] In this embodiment of the present application, if no cached data corresponding to the teaching optimization request exists, the cache status flag is determined to be "none," and this flag is passed along with the teaching optimization request to the downstream module, initiating the real-time data collection, scoring, and teaching optimization process. Initiating the real-time data collection and scoring process ensures that the system has both high responsiveness and supports personalized processing and dynamic calculation.

[0041] If no cached data is detected, a "cache status identifier" and a "business scenario identifier" are generated as the basis for subsequent data processing procedures.

[0042] The classroom improvement strategy obtained in the embodiments of the present application can be announced in the form of voice or text, which is not specifically limited in the embodiments of the present application.

[0043] Furthermore, in order to ensure the steady progress of classroom teaching and avoid interruptions to teaching. After obtaining a classroom improvement strategy through the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking provided by the embodiment of the present application, prompt information can be generated according to the classroom improvement strategy, and the prompt information can be sent to the teacher's terminal device (such as a teacher's all-in-one machine, a teacher's mobile phone) in the form of text or vibration to avoid affecting teaching directly through voice broadcast. In addition, if the user needs to announce by voice, it can be broadcast after receiving the broadcast instruction.

[0044] In an embodiment of the present application, the above-mentioned multimodal data acquisition module may include multimodal acquisition devices such as high-definition cameras, microphone arrays, and smart interactive terminals, which are used to collect voice, images, and interaction logs during classroom teaching.

[0045] In some embodiments of the present application, the first multimodal data includes: at least one of text data, sentiment score vector, video feature vector, and event timestamp and spatial location information in the current classroom.

[0046] In the embodiments of the present application, the spatial location information of the current classroom is acquired through an acquisition device, such as a high-definition camera. The spatial location information referred to herein refers to the relative position information (e.g., three-dimensional coordinates) of the image acquisition device (e.g., camera) or the indirectly associated spatial attributes of area labels (e.g., teacher area, student area). It is understood that voice data, video data, and interaction logs may contain relevant spatial location information. Therefore, it is possible to identify voice data, video data, and interaction logs to obtain richer spatial location information.

[0047] Specifically, in the embodiment of the present application, the acquisition of the first multimodal data of the current class during the classroom teaching process by the multimodal data acquisition module can be achieved in the following manner: The classroom activity data of the current class is collected through the multimodal data acquisition module; the classroom activity data includes at least: voice data, video data, interaction logs, and spatial location information in the current class; the voice data is converted into text to obtain text data; through a preset text emotion model, the emotion score vector is obtained according to the acoustic parameters of the text data and voice data; and the facial expression features and scene features of the characters in each frame of the video data are extracted, and a video feature vector is generated through the facial expression features and scene features of the characters; and the interaction logs are analyzed to determine the valid events in each time period, and the time to which the valid event belongs is set as the event timestamp.

[0048] In the absence of cached data, the embodiment of the present application starts the multimodal data acquisition module, and through the multimodal acquisition equipment (microphone array, high-definition camera and smart interactive terminal, etc.) deployed in the classroom, the voice data, video data and interaction logs during the classroom teaching process are collected and processed in real time.

[0049] From the above, it can be seen that this application requires voice data processing, video data processing, and interaction log processing to obtain the first multimodal data.

[0050] Specifically, the above-mentioned voice data processing is as follows: The collected raw speech signal is subjected to noise suppression, preferably through Wiener filtering or spectral subtraction, to enhance speech clarity. An automatic speech recognition (ASR) model based on the Transformer architecture is then used to convert the speech into structured text T, the aforementioned text data.

[0051] For sentiment analysis, this application introduces a dual-channel feature fusion mechanism. On the one hand, it extracts acoustic parameters (such as speaking speed, intonation, and pitch curve) from the speech data, and then performs sentiment classification based on the text obtained by acoustic parameters and speech conversion. It uses the BERT+BiLSTM text emotion recognition model to output the sentiment score vector. , the sentiment score vector includes sentiment categories (such as positive, neutral, negative) and sentiment intensity scores (range [0,1]).

[0052] Specifically, the above video data processing is as follows: The collected video data can be parsed by the image processing module of the CNN-Transformer hybrid architecture, and the facial expression features of the characters, including the facial expression features of the teacher and the facial expression features of the students, are extracted from each frame of the video data. Then, the MediaPipe or OpenFace model is used to identify 68 facial key points and perform emotion classification. At the same time, the object detection algorithm such as YOLOv5 is used to extract scene features (such as raising hands, standing, discussion group gathering, etc.), and then generate video feature vectors. For the subsequent construction of spatiotemporal semantic graphs. In the embodiment of the present application, all video feature vectors are organized in a frame-level temporal sequence and synchronized with the audio timeline.

[0053] Specifically, interaction log processing is as follows: Standardize the format of interaction logs from teaching platforms or terminals (including valid events such as text speeches, voting, and homework submission). Mark each record with a timestamp, extract the interaction type and content, and encode it into a structured record in a unified format to facilitate integration with voice and video data. In this embodiment of the application, the first multimodal data: ; in, For text data; is the sentiment score vector, which is used to represent the tone, speaking speed, emotion category and intensity; The event timestamp, with precision in seconds; Spatial position information, specifically the relative position information (such as three-dimensional coordinates) of the image acquisition device (such as a camera) or the area label (such as teacher area, student area); Video feature vector, integrating expression recognition and action detection information.

[0054] In an embodiment of the present application, the above-mentioned data format can be used as a unified format for the first multimodal data to obtain standardized multimodal data, thereby providing a high-quality structured foundation for the subsequent construction of a spatiotemporal semantic graph.

[0055] S102: Map each piece of first multimodal data to a node in a first spatiotemporal semantic graph, and generate the first spatiotemporal semantic graph according to a preset edge construction condition and a preset edge generation condition.

[0056] Specifically, each first multimodal data can be mapped to a node of the first spatiotemporal semantic graph; then, according to the preset edge construction conditions, the time difference, spatial Euclidean distance, and semantic similarity of the two nodes are calculated respectively; then, according to the time difference, spatial Euclidean distance, and semantic similarity, the two nodes that meet at least two preset edge generation conditions are used to construct the directed edges of the first spatiotemporal semantic graph, thereby obtaining the first semantic graph.

[0057] In the embodiment of the present application, each piece of first multimodal data in the classroom teaching activity is mapped to a node in the first spatiotemporal semantic graph, and each node includes the following attributes: ; For text data; is the sentiment score vector, which is used to represent the tone, speaking speed, emotion category and intensity; The event timestamp, with precision in seconds; Spatial position information, specifically the relative position information (such as three-dimensional coordinates) of the image acquisition device (such as a camera) or the area label (such as teacher area, student area); It is a video feature vector that integrates expression recognition and action detection information.

[0058] In the embodiment of the present application, the valid events in each time period generate corresponding node sets. .

[0059] The preset edge construction conditions in the embodiment of the present application include time constraints, space constraints, and semantic similarity constraints.

[0060] The above time constraint: If two nodes , Time difference , then the time constraint is satisfied, where is the preset time difference threshold, The value range can be set to 5~10 seconds to capture continuous interactions in a short period of time.

[0061] The above space constraints: calculate the two nodes , The spatial Euclidean distance ,like It is considered to be in the adjacent spatial area and meets the time constraint. Preset distance threshold It can be configured according to the classroom structure, and the most common setting is 1.5 meters (about the distance between two desks and seats).

[0062] The above semantic similarity constraint: calculate the two nodes , Text data in , Embedded as vectors by, for example, the BERT model , , and calculate the cosine similarity (i.e. semantic similarity): ; like , then it is considered semantically related and the similarity is preset It can be set to 0.7~0.8 to connect speeches with a logical continuation relationship.

[0063] The above preset edge generation conditions: when the node pair At least two of the above-mentioned preset edge generation conditions are met at the same time, and a line is constructed from point to Directed edges .

[0064] Through the above embodiment, the node set and edge sets By combining them, this application forms a complete spatiotemporal semantic graph: .

[0065] Furthermore, after step S102, the embodiment of the present application may also include the following method: determining the teaching scene in the classroom teaching process, wherein the teaching scene includes at least one of lectures, question-answering and discussions; then clustering the first spatiotemporal semantic graph according to the teaching scene to obtain a semantic subgraph of each teaching scene, and setting each semantic subgraph as the first spatiotemporal semantic graph.

[0066] In the embodiment of the present application, in order to enhance the modeling capability of different teaching segments (such as lectures, question-answering, and discussions), the first spatiotemporal semantic graph can be constructed based on the semantic features and interaction density of the nodes. The division is as follows: (1) Using a partitioning method based on embedding clustering; (2) Each semantic subgraph represents a relatively independent teaching scenario (such as “group discussion 1” or “questioning and interactive paragraph”); (3) Each semantic subgraph maintains the original edge structure and serves as the input unit for the subsequent scoring stage.

[0067] This first spatiotemporal semantic graph and its semantic subgraphs not only retain the temporal and spatial information of events, but also integrate semantic and emotional attributes, providing structured input for subsequent graph neural networks and supporting in-depth classroom behavior modeling and status assessment.

[0068] "Thematic subgraph division" is to automatically decompose the spatiotemporal semantic graph with mixed structure in an entire class into several semantically concentrated and temporally continuous fragmented semantic graphs, so as to more accurately analyze the performance of each teaching link, support more fine-grained and personalized scoring and suggestions, and further improve the teaching quality.

[0069] S103: Input the first spatiotemporal semantic graph into a preset graph neural network to obtain a first dynamic scoring vector in the course teaching process.

[0070] In the embodiment of the present application, the first spatiotemporal semantic graph constructed in the above step S102 is Input the pre-set Graph Neural Network (GNN) and use the Graph Convolutional Network (GCN) to extract features and fuse information on the first spatiotemporal semantic graph, and finally generate the first dynamic scoring vector used to represent the classroom status, as shown below: 1. Initial feature vector embedding, as follows: For each node in the first spatiotemporal semantic graph , first its attributes Perform unified embedding processing: 1) Text Data Extract semantic vectors through the BERT model; 2) Sentiment score vector and video feature vector Direct splicing and projection; 3) Event timestamp and spatial location information Encoded as sine-cosine temporal features and spatial position vectors; Finally generate the initial node vector , its dimension is usually set to 128~256 dimensions as the input of GCN.

[0071] in, The specific operations include: Converted to a 768-dimensional vector through the BERT model; sentiment score vector (3D), video feature vector (128 dimensions), timestamp t (encoded as a 2-dimensional sine and cosine vector), and spatial position s (encoded as a 3-dimensional coordinate vector) are concatenated to form a total dimension of: 768 (T) + 3 ( )+128( )+2(t)+3(s)=904-dimensional original feature vector is projected to 256 dimensions through the fully connected layer, that is, .

[0072] 2. Graph convolutional network structure and parameter settings are as follows: The graph neural network in the embodiment of the present application can adopt a three-layer graph convolution structure (3-layer GCN) with the number of layers L=3, which is used to capture the multi-order adjacency information of nodes in the spatiotemporal semantic graph.

[0073] The feature update formula for each layer is: ; in: is the feature representation of node v in layer l; is the set of adjacent nodes of node v; 、 is the weight matrix and bias term of the lth layer; As the activation function, the embodiment of the present application can use the ReLU (Rectified Linear Unit) activation function, which has both nonlinear expression ability and training stability.

[0074] In an embodiment of the present application, the node vector dimension of each layer output can be set as follows: first layer: input dimension 128, output dimension 128; second layer: input dimension 128, output dimension 64; third layer: input dimension 64, output dimension 32.

[0075] In the embodiment of the present application, high-order semantic structural features of nodes can be effectively extracted through layer-by-layer dimensionality reduction and compression.

[0076] 3. Graph-level pooling and score mapping, as follows: After completing node-level feature learning, global feature aggregation is performed on the entire spatiotemporal semantic graph to generate a unified graph-level table. The calculation using the Global Average Pooling strategy is as follows: ; in Calculation logic: the final feature vector of all nodes (L is the number of GCN layers, such as 3 layers, output 32 dimensions) take the average value, where is the total number of nodes. Example: If there are 10 nodes, each node is a 32-dimensional vector, then It is the average value of the corresponding dimensions of the 10 vectors, which is still 32 dimensions.

[0077] This graph-level vector comprehensively reflects the multimodal state information of the entire classroom teaching process and has good discriminative ability.

[0078] In the embodiment of the present application, a fully connected network (Dense Layer) is used to connect Mapped to the first dynamic score vector: ; in (or more dimensions), is a 32×3 weight matrix (the 32-dimensional Mapped to a 3-dimensional rating vector), is a 3D bias term, trained through supervised learning (with manually annotated ratings as labels).

[0079] Each component corresponds to the following evaluation dimensions: (1) teacher teaching quality rating; (2) student participation rating; (3) classroom atmosphere rating.

[0080] In the embodiment of the present application, the first dynamic scoring vector can be immediately stored in a local cache for subsequent call by an improvement suggestion generation module, or can be pushed to a teaching management platform through an interface.

[0081] The graph neural network module provided in the embodiment of the present application can adopt a supervised learning mechanism and set the training goal to minimize the residual between the model scoring result and the true score or historical high-quality score. The true score can be marked by a teacher or administrator, or provided by a predefined high-quality classroom case library.

[0082] S104 : Generate a classroom improvement strategy based on a preset strategy network and according to the first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector.

[0083] In an embodiment of the present application, the strategy network corresponding to the first spatiotemporal semantic graph can be first obtained; and the deviation between the calculated first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector can be set as the current classroom state; the current classroom state can be input into the strategy network corresponding to the first spatiotemporal semantic graph to generate a classroom improvement strategy.

[0084] It can be seen from the above embodiments that the first spatiotemporal semantic graph in the embodiments of the present application can be a divided semantic subgraph. At this time, more targeted strategy networks can be adopted respectively, thereby further improving the quality of classroom teaching optimization.

[0085] After obtaining the first dynamic score vector through step S103 Then, compare it with the target score vector in the history high-quality classroom case library The comparison is then performed, and the current classroom state representation st is calculated based on the deviation between the two, and input into the preset policy network to generate a targeted classroom improvement strategy, as shown below: 1. State representation and scoring deviation modeling, as follows: Classroom Status It is defined as the deviation between the first dynamic score vector and the target score vector: ; in, Deviation in the quality of teaching for teachers; for student engagement bias; The classroom atmosphere is biased.

[0086] For example, if = [80, 75, 85] (teaching quality, participation, atmosphere), = [90, 85, 90], then =[10,10,5].

[0087] Among them, the quality of teachers' teaching is calculated through weighted factors such as the completeness of the teaching process (such as the coverage of knowledge points), the clarity of language expression (the fluency score of speech recognition), and the logic of blackboard writing / PPT (the text semantic coherence score). The weights can be set by experts (such as 40% for coverage, 30% for clarity, and 30% for logic).

[0088] Student engagement: Quantified based on speaking frequency (number of interactions per 5 minutes), question quality (question relevance score), and physical activity (number of hand raises / nods identified by video), for example, engagement = (speaking frequency × 0.5 + question quality × 0.3 + physical activity × 0.2).

[0089] Classroom atmosphere: This factor combines the student emotion score vector (percentage of positive emotions), group interaction intensity (percentage of group discussion time), etc. For example, atmosphere score = proportion of positive emotions × 0.6 + proportion of discussion time × 0.4.

[0090] Deviation calculation: The deviation of each dimension is the difference between the target value and the actual value, that is, .

[0091] Regarding the teacher teaching quality scoring method (used to calculate Δ_teach), the teacher teaching quality scoring must first quantify each sub-item (teaching process completeness, language clarity, blackboard / PPT logic), and then use weighted factors to obtain the total score, as follows: ① Completeness of teaching links (coverage of knowledge points) Data sources: the list of knowledge points preset in the course syllabus, the voice transcription text or PPT content of the teacher’s lecture.

[0092] Scoring method: Calculate the ratio of the number of knowledge points actually covered by the teacher (denoted as K) to the total number of knowledge points required by the syllabus (denoted as K_total). In other words, coverage score = (K / K_total) × 100. For example, if the syllabus requires 10 knowledge points and the teacher actually covers 8, the coverage score is (8 / 10) × 100 = 80.

[0093] ② Language expression clarity (speech recognition fluency) Data source: Teacher's voice signal collected by microphone, text converted by ASR (automatic speech recognition) model, and recognition log.

[0094] Scoring method: Calculated based on speech recognition accuracy and coherence: Clarity score = (Accuracy × 0.7 + Coherence × 0.3) × 100, where Accuracy = number of correctly recognized words / total number of words (e.g., if 95 out of 100 words are correctly recognized, the accuracy is 0.95), and Coherence = the percentage of sentences without pauses or stutters (e.g., if 8 out of 10 sentences flow smoothly, the coherence is 0.8). For example, if the accuracy is 0.95 and the coherence is 0.8, the Clarity score = (0.95 × 0.7 + 0.8 × 0.3) × 100 = 90.5.

[0095] ③ Blackboard / PPT logic (text semantic coherence) Data source: OCR text of blackboard photos and text content of PPT.

[0096] Scoring method: The semantic similarity of adjacent content is calculated using a text semantic model (BERT) and the average is calculated: Logical score = average semantic similarity × 100. For example, if the semantic similarities of three consecutive chapters in a PPT are 0.8, 0.9, and 0.7, respectively, the average similarity is 0.8, and the logical score is 80.

[0097] ④Total score of teacher teaching quality Calculated according to the weights set by experts (such as coverage 40%, clarity 30%, logic 30%): Teacher teaching quality score = coverage score × 0.4 + clarity score × 0.3 + logic score × 0.3.

[0098] Regarding the student engagement scoring method (used to calculate Δ_engage), student engagement requires quantifying and weighting the frequency of speaking, the quality of questions asked, and the degree of physical activity, as follows: ① Speaking frequency (number of interactions per 5 minutes) Data source: ASR recognition results of students' voices and text speech records from classroom interactive terminals.

[0099] Scoring method: Count the total number of times a student actively speaks (including voice and text) within every 5 minutes and map it to a score of 0 to 100: Speech frequency score = min(actual number of times / preset number of times achieved × 100, 100). For example, if the preset number of times achieved within 5 minutes is 5, and the student actually speaks 3 times, the score = 3 / 5 × 100 = 60.

[0100] ② Question quality (relevance of questions to the course) Data sources: text content of students’ questions (voice transcription or text input), and labels of course knowledge points.

[0101] Scoring: A text matching model (TFIDF or BERT) is used to calculate the relevance score between the question and the course content (ranging from 0 to 1). This score is then converted to a percentage: Question Quality Score = Relevance Score × 100. For example, if a student's question, "How do I find the maximum value of a quadratic function?" has a relevance score of 0.9 to the course content, "Properties of Quadratic Functions," the score would be 90.

[0102] ③ Physical activity (number of times hands are raised / nodded) Data source: Video streams from classroom cameras and student body movements identified by the object detection model (YOLOv5).

[0103] Scoring method: Count the total number of times students raise their hands and nod their heads every 5 minutes and map it to a score of 0 to 100: Physical activity score = min(actual number of actions / preset number of actions achieved) × 100, 100. For example, if the preset number of actions achieved within 5 minutes is 10 and the actual number is 6, the score is 60.

[0104] ④Total score of student participation Calculated by weight (speaking frequency 50%, questioning quality 30%, physical activity 20%): Student participation score = speaking frequency score × 0.5 + questioning quality score × 0.3 + physical activity score × 0.2 Regarding the classroom atmosphere scoring method (used to calculate Δ_mood), the classroom atmosphere is quantified based on student emotion scores and group interaction intensity, as follows: ① Student emotion score vector (percentage of positive emotions) Data source: Video recognition results of students’ facial expressions (classified as positive, neutral, and negative using the OpenFace model).

[0105] Scoring method: Calculate the proportion of students with positive emotions to the total number of students: Positive Emotion Percentage = Number of Students with Positive Emotions / Total Number of Students. For example, if 21 out of 30 students express positive emotions, the Positive Emotion Percentage = 0.7.

[0106] ② Group interaction intensity (percentage of group discussion time) Data source: The duration of group discussion scenes (such as gatherings of multiple people and physical interactions) detected in the video stream.

[0107] Scoring method: Calculate the proportion of group discussion time to total class time: Discussion time percentage = total discussion time / total class time. For example, if the discussion time is 15 minutes in a 45-minute class, the percentage is 15 / 45, which is ≈ 0.333.

[0108] ③Total score of classroom atmosphere Calculate the classroom atmosphere score based on the weights (positive emotions account for 60% and discussion time accounts for 40%): Positive emotions × 0.6 × 100 + Discussion time × 0.4 × 100. For example, if positive emotions account for 0.7 and discussion time accounts for 0.333, the score would be 0.7 × 60 + 0.333 × 40 ≈ 42 + 13.3 = 55.3.

[0109] The scores for each dimension are obtained through the process of "sub-item quantification → percentage conversion → weighted summary", and finally form Score_Vector (teaching quality, engagement, atmosphere), which is then compared with Score_Target to obtain deviations Δ_teach, Δ_engage, and Δ_mood, which provide input for the strategy network.

[0110] The above deviations reflect the degree of deficiency of the current classroom in key evaluation dimensions, which serve as the input state of the preset strategy network.

[0111] 2. Policy network structure and training method, as follows: In the embodiment of the present application, the policy gradient method (Policy Gradient) can be used for policy optimization. The above policy network Output in state Select a classroom improvement strategy The probability distribution of . The parameter update formula is: ; in: is the policy network parameter; is the learning rate (such as 0.001); is the actual reward value; It is the benchmark value used to reduce variance and improve stability. For the policy network in state Select Action The log-odds probability of the parameter gradient.

[0112] 3. Reward function design, as follows: In the embodiment of this application, the reward value The design is based on the improvement effect of teaching score, and mainly considers the new dynamic score vector after the implementation of the suggestion (i.e., the second dynamic rating vector) and the change in historical deviation: ; in: is the original score deviation; For the recommendation to implement the scoring bias; Improve the weight factor for scoring and control the intensity of positive incentives; Cost factors (such as system resources required for complex feedback); To generate classroom improvement strategies System cost functions (such as speech synthesis time or bandwidth usage); This reward function can effectively measure the positive teaching effect brought by the suggestions while constraining resource consumption.

[0113] 4. Benchmark value The setting method is as follows: Baseline value It can be set dynamically in the following ways: (1) Moving average method: take the average of recent (e.g. past 20) reward values ​​and set it as the current benchmark , used to smooth training fluctuations; (2) Historical optimal value comparison method: set it as the average reward after the recommended execution under similar courses.

[0114] The processing of the above embodiment can stabilize the gradient estimation and improve the strategy convergence efficiency.

[0115] 5. Personalized suggestion generation strategy, as follows: Generate different types of improvement suggestions based on the current business scenario: (1) Fixed feedback scenario: directly retrieve suggested content that matches the deviation pattern from a preset template library, suitable for low-interaction devices or standard courses; (2) Local feedback scenario: combining template suggestions with user behavior variables (e.g., weak teacher intonation, sparse student interaction), and generating customized voice broadcast content through the local speech synthesis module; (3) Cloud feedback scenario: The system calls the cloud-based large model service to generate recommended content, and outputs it through streaming synthesis technology (such as Tacotron2 + WaveGlow), while caching the results to optimize the response speed.

[0116] Through the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking provided by the embodiment of the present application, the final classroom improvement strategy will be broadcast on the terminal in the form of audio or text for teachers to improve the classroom.

[0117] In some embodiments of the present application, after step S104, the present application may further include the following steps: when it is detected that the teacher executes the classroom improvement strategy, the system will start a new round of data collection in response to the improvement instructions based on the classroom improvement strategy feedback; obtain second multimodal data in the classroom teaching process within a preset time period, wherein the second multimodal data includes: text data, sentiment score vector, at least one of the video feature vector, and event timestamp and spatial location information in the current classroom, etc.; map each second multimodal data to a node in the second spatiotemporal semantic graph, and generate a second spatiotemporal semantic graph based on preset edge construction conditions and preset edge generation conditions; input the second spatiotemporal semantic graph into the graph neural network to generate a second dynamic scoring vector; update the strategy network according to the second dynamic scoring vector and the target score corresponding to the second dynamic scoring vector.

[0118] It is understandable that the specific implementation methods of obtaining the second dynamic scoring vector in the above embodiment can all refer to the above steps S101 to S103, which will not be described in detail here.

[0119] In an embodiment of the present application, after detecting the implementation of the classroom improvement strategy, an improvement instruction is generated. At this time, the second multimodal data of the classroom teaching process within a period of time (i.e., a preset time period, such as 5 minutes) can be re-acquired, and then a second spatiotemporal semantic graph is constructed and a dynamic scoring vector (i.e., a second dynamic evaluation vector) after the implementation of the classroom improvement strategy is obtained based on the second spatiotemporal semantic graph. The strategy network is updated based on the second dynamic scoring vector and the target score corresponding to the second dynamic scoring vector, thereby realizing small-batch self-supervised incremental training, real-time updating of the strategy network, and continuous optimization of model performance.

[0120] The above target scores are preset benchmark values, which are typical scoring data extracted from the history high-quality classroom case library, or manually preset standard values.

[0121] In the embodiment of the present application, a small batch self-supervised online learning mechanism can be used to regularly use new data to perform lightweight incremental training on the graph neural network and the policy network, wherein: the update frequency can be set to trigger online training every time a complete classroom record after 20 suggestions are executed; the batch size is set to 8 to 16 records by default to avoid model drift caused by large batches; the learning rate control is to use a small learning rate in the online update phase. , to reduce the risk of catastrophic forgetting; data buffering strategy: can retain recent 50 (e.g., 50) high-quality labeled samples constitute the retraining samples to assist in the mixed training of new and old data; network freezing layer setting: during the online training process, freeze the first few layers of parameters in the graph neural network and policy network, and only update the high-level feature fusion module and output layer to improve learning efficiency.

[0122] During the update process of the embodiment of the present application, the neural network parameters are iterated in the following manner: 1) The graph neural network uses the minimization of the score residual as the supervision goal to optimize the feature extraction accuracy; 2) The policy network uses the cumulative reward value as the objective function to optimize the strategy; 3) All update operations are completed asynchronously in the background without affecting the front-end real-time feedback process.

[0123] In some embodiments of the present application, a new dynamic scoring vector is generated based on the updated graph neural network. , the following dynamic information can be displayed through the graphical user interface (GUI): scoring indicator trend chart: displays the curve of scoring dimensions (teaching quality, participation, atmosphere, etc.) changing over time; suggestion adoption effect chart: displays the comparison of scores before and after the implementation of the suggestion, and quantifies the improvement; feedback response contour chart: identifies the effectiveness of each suggestion, which suggestions have produced positive effects and which have not yet met expectations, to assist teachers in optimizing teaching strategies; model status prompt: including the current neural network version, the time of the last online update, the cumulative learning rounds, etc.

[0124] In an embodiment of the present application, the user can choose through the interface whether to mark the current data as high-confidence feedback, and the system will adjust the training data priority accordingly to strengthen the learning weight of the corresponding strategy.

[0125] The policy network provided in the embodiment of the present application can adopt unsupervised reinforcement learning based on reward feedback. Its reward signal is calculated according to the change in classroom score after the implementation of the improved strategy. Specifically, the difference in score deviation before and after the implementation of the strategy is used as a positive reward, and combined with system resource consumption as a negative weight to form an overall reward value. The reward signal is input into the policy network as training feedback to optimize the generation strategy. That is, the reward signal guides the policy network to learn the "high score improvement-low resource consumption" generation strategy, realizing the automation and personalization of teaching improvement.

[0126] For example, the classroom improvement strategy for the first class was "adding a 5-minute discussion session". After the implementation of the classroom improvement strategy, the score of student interaction increased from 75 to 85, and the system recorded: reward = +10 (significantly effective); the classroom improvement strategy for the second class was "adding image display"; after implementation, the score of teaching quality increased from 88 to 89, with a small change; reward = +1 (weak effect); the classroom improvement strategy for the third class was "adding a five-minute question session". After the implementation of the classroom improvement strategy, the teaching quality increased from 75 to 73, reward = -2, and no effect.

[0127] like Figure 2 As shown, the present application can first detect whether there is a valid cache (i.e., there is cache data corresponding to the teaching optimization request) through the cache detection module; if so, the classroom improvement strategy can be broadcast directly through the cache data; if not, the first multimodal data can be obtained through the multimodal data acquisition module, and then a first spatiotemporal semantic graph is constructed based on the first multimodal data, and then the first spatiotemporal semantic graph is dynamically scored through the graph neural network to obtain a first dynamic scoring vector, and then a classroom improvement strategy is generated according to the strategy network and broadcast, and then when it is detected that the teacher has executed the classroom improvement strategy, in response to the improvement instruction based on the classroom improvement strategy feedback, a second dynamic evaluation vector is obtained through the adaptive feedback mechanism, and then the neural network parameters are updated and visualized through the second dynamic evaluation vector.

[0128] It can be understood that the above-mentioned improvement instructions are used to indicate that the teacher is detected to be implementing a classroom improvement strategy.

[0129] Compared with the existing technology, in the solution corresponding to the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking provided by this application, a first spatiotemporal semantic graph reflecting the teaching process is constructed through the first multimodal data in the classroom teaching process. In this technology, the first spatiotemporal semantic graph is subjected to data fusion and feature extraction through a graph neural network to obtain a first dynamic scoring vector in the classroom teaching process. Then, through a preset strategy network, a corresponding classroom improvement strategy is generated according to the first dynamic scoring vector and the corresponding target score to guide the adjustment of classroom teaching content. Through the above solution, real-time perception and dynamic scoring of the classroom teaching process are realized, and more accurate classroom improvement strategies are obtained, so that on the basis of improving the real-time, comprehensiveness and guidance of course teaching evaluation, the teaching plan can be adjusted in real time, the teaching content can be optimized, and the quality of classroom teaching can be improved.

[0130] Based on the above embodiment, the present application also provides a smart terminal, whose principle block diagram can be as follows: Figure 3 As shown. The above-mentioned intelligent terminal includes a processor, a memory, a network interface and a display screen connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, any one of the steps of the above-mentioned dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking is implemented. The display screen of the intelligent terminal can be a liquid crystal display or an electronic ink display.

[0131] Those skilled in the art will understand that Figure 3 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the smart terminal to which the solution of the present application is applied. The specific smart terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0132] In one embodiment, a smart terminal is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of any one of the dynamic classroom optimization methods based on real-time spatiotemporal semantic graph tracking provided in the embodiments of the present application are implemented.

[0133] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the dynamic classroom optimization methods based on real-time spatiotemporal semantic graph tracking provided in the embodiment of the present application are implemented.

[0134] It should be understood that the serial numbers of the steps in the above embodiments do not imply the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0135] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0136] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0137] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] In the embodiments provided herein, it should be understood that the disclosed systems / terminal devices and methods can be implemented in other ways. For example, the system / terminal device embodiments described above are merely illustrative. For example, the division of the modules or units described above is merely a logical functional division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another system, or omitting or not implementing certain features.

[0139] If the above-mentioned integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present application can also implement all or part of the process steps in the above-mentioned method embodiments by using a computer program to instruct the relevant hardware. The above-mentioned computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The above-mentioned computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The above-mentioned computer-readable medium can include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, electrical signal, and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0140] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking, characterized in that: The method comprises: In response to the teaching optimization request, obtaining first multimodal data of the current class during the classroom teaching process; Mapping each piece of the first multimodal data into a node in a first spatiotemporal semantic graph, and generating the first spatiotemporal semantic graph based on a preset edge construction condition and a preset edge generation condition; Inputting the first spatiotemporal semantic graph into a preset graph neural network to obtain a first dynamic scoring vector during classroom teaching; Based on a preset strategy network and according to the first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector, a classroom improvement strategy is generated.

2. The dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking according to claim 1 is characterized in that: The step of obtaining first multimodal data of the current class during the classroom teaching process in response to the teaching optimization request includes: In response to the teaching optimization request, starting a preset cache detection module; Through the cache detection module, query whether there is cache data of the teaching optimization request; If the cached data of the teaching optimization request does not exist, the multimodal data acquisition module is started to obtain the first multimodal data of the current class during the classroom teaching process through the multimodal data acquisition module.

3. The dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking according to claim 2 is characterized in that: The method further comprises: If the cached data of the teaching optimization request exists, the pre-improvement strategy in the cached data is set as the class improvement strategy of the current class and announced.

4. The dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking according to claim 2 is characterized in that: The first multimodal data includes: at least one of text data, a sentiment score vector, a video feature vector, an event timestamp, and spatial location information in the current classroom; The acquiring, by the multimodal data acquisition module, the first multimodal data of the current class during the classroom teaching process includes: The multimodal data acquisition module is used to collect classroom activity data of the current class, wherein the classroom activity data includes: voice data, video data, interaction logs, and spatial location information in the current class; wherein the spatial location information is obtained based on the image acquisition device in the multimodal acquisition module; Converting the voice data into text to obtain the text data; Obtaining the emotion score vector according to the acoustic parameters of the text data and the speech data using a preset text emotion model; and Extracting facial expression features and scene features of each frame of the video data, and generating a video feature vector based on the facial expression features and scene features; and The interaction log is analyzed to determine valid events in each time period, and the time to which the valid event belongs is set as the event timestamp.

5. The dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking according to claim 1 is characterized in that: Mapping each piece of the first multimodal data into a node in a first spatiotemporal semantic graph, and generating the first spatiotemporal semantic graph based on a preset edge construction condition and a preset edge generation condition, includes: Mapping each piece of the first multimodal data into a node of a first spatiotemporal semantic graph; According to the preset edge construction conditions, respectively calculating the time difference, spatial Euclidean distance, and semantic similarity of the two nodes; According to the time difference, the spatial Euclidean distance, and the semantic similarity, constructing a directed edge of the first spatiotemporal semantic graph from two of the nodes that meet at least two of the preset edge generation conditions to obtain the first spatiotemporal semantic graph; The preset edge generation conditions include: the time difference is less than or equal to a preset time difference threshold, the spatial Euclidean distance is less than or equal to a preset distance threshold, and the semantic similarity is greater than or equal to a preset similarity.

6. The dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking according to claim 5 is characterized in that: After constructing a directed edge of the first spatiotemporal semantic graph from two nodes that meet at least two preset edge generation conditions according to the time difference, the spatial Euclidean distance, and the semantic similarity to obtain the first spatiotemporal semantic graph, the method further includes: Determining a teaching scenario in the classroom teaching process, wherein the teaching scenario includes at least one of lectures, question-answering, and discussions; The first spatiotemporal semantic graph is clustered according to the teaching scene to obtain a semantic subgraph of each teaching scene, and each semantic subgraph is set as a first spatiotemporal semantic graph.

7. The dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking according to claim 1 is characterized in that: The method of generating a classroom improvement strategy based on a preset strategy network and according to the first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector includes: Obtaining the policy network corresponding to the first spatiotemporal semantic graph; and Calculating a deviation between the first dynamic scoring vector and a target score corresponding to the first dynamic scoring vector, and setting the deviation as the current class state; The current classroom state is input into the strategy network corresponding to the first spatiotemporal semantic graph to generate the classroom improvement strategy.

8. The dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking according to claim 1 is characterized in that: After generating the classroom improvement strategy, the method further includes: In response to an improvement instruction based on the classroom improvement strategy feedback, obtaining second multimodal data of the classroom teaching process within a preset time period; wherein the improvement instruction is used to indicate that the teacher is detected to be implementing the classroom improvement strategy; Mapping each piece of the second multimodal data into a node in a second spatiotemporal semantic graph, and generating the second spatiotemporal semantic graph based on a preset edge construction condition and a preset edge generation condition; Inputting the second spatiotemporal semantic graph into the graph neural network to generate a second dynamic scoring vector; The policy network is updated according to the second dynamic scoring vector and a target score corresponding to the second dynamic scoring vector.

9. An intelligent terminal, characterized in that: The intelligent terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking as described in any one of claims 1 to 8.

Images