Dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking

Abstract

The application discloses a dynamic classroom optimization method based on real-time space-time semantic graph tracking, relates to the field of artificial intelligence, and comprises the following steps: in response to a teaching optimization request, acquiring first multi-modal data of a current classroom in a classroom teaching process; mapping each piece of first multi-modal data into a node in a first space-time semantic graph, and generating the first space-time semantic graph based on a preset edge construction condition and a preset edge generation condition; then inputting the first space-time semantic graph into a preset graph neural network to obtain a first dynamic score vector in the classroom teaching process; and then generating a classroom improvement strategy based on a preset strategy network and according to the first dynamic score vector and a target score corresponding to the first dynamic score vector, so that real-time perception and dynamic scoring of the classroom teaching process are realized, a more accurate classroom improvement strategy is obtained, and the real-time and comprehensiveness of course teaching evaluation are improved, and the teaching scheme can be adjusted in real time, and the classroom teaching quality is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking. Background Technology

[0002] Classroom teaching evaluation helps teachers better understand students' learning progress, thereby optimizing teaching plans and helping students achieve better learning outcomes. It plays a crucial role in classroom teaching. In recent years, classroom teaching evaluation has gradually evolved from traditional methods such as manual observation and questionnaires to automation and intelligence. It utilizes devices such as cameras, microphones, and interactive terminals to collect audio, video, and behavioral data from the classroom, enabling the perception and analysis of the teaching process.

[0003] Current classroom teaching evaluation technologies are, on the one hand, mostly "post-assessment," failing to provide real-time feedback during the teaching process; on the other hand, their data fusion strategies are crude, often neglecting the deep relationships between time, space, and semantics, thus failing to provide a comprehensive understanding of the classroom teaching process. Based on existing classroom teaching evaluation methods, it is difficult to adjust teaching plans and optimize teaching content in real time, impacting the quality of classroom teaching.

[0004] Therefore, the relevant technologies still need to be improved and developed. Summary of the Invention

[0005] The main purpose of this application is to provide a dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking, which aims to solve the technical problem in related technologies that it is difficult to adjust teaching plans and optimize teaching content in real time based on classroom teaching evaluation.

[0006] To achieve the above objectives, the first aspect of this application provides a dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking, wherein the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking includes:

[0007] In response to the teaching optimization request, acquire the first multimodal data of the current classroom teaching process;

[0008] Each piece of the first multimodal data is mapped to a node in the first spatiotemporal semantic graph, and the first spatiotemporal semantic graph is generated based on preset edge construction conditions and preset edge generation conditions;

[0009] The first spatiotemporal semantic graph is input into a preset graph neural network to obtain the first dynamic scoring vector in the classroom teaching process;

[0010] Based on a preset strategy network, and according to the first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector, a classroom improvement strategy is generated.

[0011] Optionally, the step of responding to the teaching optimization request by acquiring the first multimodal data of the current classroom teaching process includes:

[0012] In response to the teaching optimization request, the preset cache detection module is activated;

[0013] The cache detection module is used to check whether there is cached data for the teaching optimization request.

[0014] If the cached data for the teaching optimization request does not exist, the multimodal data acquisition module is activated to obtain the first multimodal data of the current classroom during the classroom teaching process.

[0015] Optionally, the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking further includes:

[0016] If the cached data for the teaching optimization request exists, the pre-improvement strategy in the cached data is set as the classroom improvement strategy for the current classroom and published.

[0017] Optionally, the first multimodal data includes at least one of text data, sentiment score vector, and video feature vector, as well as event timestamp and spatial location information in the current classroom;

[0018] The step of acquiring the first multimodal data of the current classroom during the classroom teaching process through the multimodal data acquisition module includes:

[0019] The multimodal data acquisition module collects classroom activity data for the current classroom, including: voice data, video data, interaction logs, and spatial location information within the current classroom; wherein the spatial location information is obtained from the image acquisition device in the multimodal acquisition module.

[0020] The speech data is converted into text to obtain the text data;

[0021] The emotion score vector is obtained by using a pre-defined text emotion model based on the acoustic parameters of the text data and the speech data; and

[0022] Extract facial expression features and scene features from each frame of the video data, and generate a video feature vector using the facial expression features and scene features; and

[0023] The interaction logs are analyzed to determine the valid events within each time period, and the time of each valid event is set as the event timestamp.

[0024] Optionally, the step of mapping each piece of the first multimodal data to a node in the first spatiotemporal semantic graph, and generating the first spatiotemporal semantic graph based on preset edge construction conditions and preset edge generation conditions, includes:

[0025] Each piece of the first multimodal data is mapped to a node in the first spatiotemporal semantic graph;

[0026] Based on the preset edge construction conditions, calculate the time difference, spatial Euclidean distance, and semantic similarity between the two nodes respectively;

[0027] Based on the time difference, the spatial Euclidean distance, and the semantic similarity, directed edges are constructed on two nodes that meet at least two of the preset edge generation conditions to form the first spatiotemporal semantic graph, so as to obtain the first spatiotemporal semantic graph.

[0028] The preset edge generation conditions include: the time difference is less than or equal to a preset time difference threshold, the spatial Euclidean distance is less than or equal to a preset distance threshold, and the semantic similarity is greater than or equal to a preset similarity.

[0029] Optionally, after constructing directed edges for the first spatiotemporal semantic graph from two nodes that meet at least two of the preset edge generation conditions based on the time difference, the spatial Euclidean distance, and the semantic similarity, the method further includes:

[0030] The teaching scenarios in the classroom teaching process are determined, and the teaching scenarios include at least one of lecturing, Q&A, and discussion;

[0031] The first spatiotemporal semantic graph is clustered and divided according to the teaching scenario to obtain a semantic subgraph for each teaching scenario, and each semantic subgraph is set as the first spatiotemporal semantic graph.

[0032] Optionally, the step of generating classroom improvement strategies based on a preset strategy network and according to the first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector includes:

[0033] Obtain the policy network corresponding to the first spatiotemporal semantic graph; and

[0034] Calculate the deviation between the first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector, and set it as the current classroom state;

[0035] The current classroom state is input into the policy network corresponding to the first spatiotemporal semantic graph to generate the classroom improvement strategy.

[0036] Optionally, after generating classroom improvement strategies, the method further includes:

[0037] In response to an improvement instruction based on feedback from the classroom improvement strategy, second multimodal data of the classroom teaching process within a preset time period is acquired; wherein, the improvement instruction is used to indicate that the teacher has been detected to be implementing the classroom improvement strategy;

[0038] Each piece of the second multimodal data is mapped to a node in the second spatiotemporal semantic graph, and the second spatiotemporal semantic graph is generated based on preset edge construction conditions and preset edge generation conditions;

[0039] The second spatiotemporal semantic graph is input into the graph neural network to generate a second dynamic scoring vector;

[0040] The strategy network is updated based on the second dynamic scoring vector and the target score corresponding to the second dynamic scoring vector.

[0041] Secondly, this application provides a terminal device, the smart terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements any of the steps of the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracing.

[0042] A third aspect of this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any one of the described dynamic classroom optimization methods based on real-time spatiotemporal semantic graph tracing.

[0043] To address the aforementioned issues, in this application's solution, in response to a teaching optimization request, the first multimodal data of the current classroom teaching process is acquired; each piece of first multimodal data is mapped to a node in a first spatiotemporal semantic graph, and a first spatiotemporal semantic graph is generated based on preset edge construction conditions and preset edge generation conditions. Then, the first spatiotemporal semantic graph is input into a preset graph neural network to obtain a first dynamic scoring vector during the classroom teaching process; then, based on a preset policy network and according to the first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector, a classroom improvement strategy is generated.

[0044] Compared with existing technologies, the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking provided in this application constructs a first spatiotemporal semantic graph reflecting the teaching process using first multimodal data during classroom teaching. Based on this technology, a graph neural network is used to perform data fusion and feature extraction on the first spatiotemporal semantic graph to obtain a first dynamic scoring vector during classroom teaching. Then, through a pre-set strategy network, a corresponding classroom improvement strategy is generated based on the first dynamic scoring vector and the corresponding target score to guide adjustments to classroom teaching content. This approach achieves real-time perception and dynamic scoring of the classroom teaching process, resulting in more accurate classroom improvement strategies. Therefore, it improves the real-time nature, comprehensiveness, and guidance of course teaching evaluation, enabling real-time adjustments to teaching plans, optimization of teaching content, and enhancement of classroom teaching quality. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating a dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking, as provided in an embodiment of this application.

[0047] Figure 2 This is another flowchart illustrating a dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking provided in an embodiment of this application;

[0048] Figure 3 This is a block diagram illustrating the internal structure of a smart terminal provided in an embodiment of this application. Detailed Implementation

[0049] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0050] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0051] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0052] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0053] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to classification." Similarly, the phrases "if determined" or "if classified to [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once classified to [the described condition or event]," or "in response to classification to [the described condition or event]."

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0055] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0056] With the advancement of science and technology, classroom teaching evaluation is gradually shifting from traditional methods such as manual observation and questionnaires towards automation and intelligence. Current research utilizes devices like cameras, microphones, and interactive terminals to collect audio, video, and behavioral data from the classroom to perceive and analyze the teaching process. For example, intelligent teaching systems integrating multimodal behavioral perception technologies propose using deep learning models to analyze indicators such as student-student interaction, facial expressions, and speaking frequency; and employing strategies that integrate sentiment analysis and speech recognition to construct classroom teaching evaluation models, enhancing the ability to capture student participation. While these methods have improved the objectivity and automation of classroom evaluation to some extent, significant shortcomings remain. On the one hand, existing technologies are mostly "post-evaluation," unable to provide real-time feedback during the classroom process; on the other hand, data fusion strategies are often crude, frequently neglecting the deep relationships between temporal, spatial, and semantic dimensions, making it difficult to comprehensively depict the state of classroom interaction.

[0057] Based on existing classroom teaching evaluations, it is difficult to adjust teaching plans and optimize teaching content in real time according to classroom teaching evaluations, which affects the quality of classroom teaching.

[0058] To address the aforementioned issues, in this application's solution, in response to a teaching optimization request, the first multimodal data of the current classroom teaching process is acquired; each piece of first multimodal data is mapped to a node in a first spatiotemporal semantic graph, and a first spatiotemporal semantic graph is generated based on preset edge construction conditions and preset edge generation conditions. Then, the first spatiotemporal semantic graph is input into a preset graph neural network to obtain a first dynamic scoring vector during the classroom teaching process; then, based on a preset policy network and according to the first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector, a classroom improvement strategy is generated.

[0059] Compared with existing technologies, the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking provided in this application constructs a first spatiotemporal semantic graph reflecting the teaching process using first multimodal data during classroom teaching. Based on this technology, a graph neural network is used to perform data fusion and feature extraction on the first spatiotemporal semantic graph to obtain a first dynamic scoring vector during classroom teaching. Then, through a pre-set strategy network, a corresponding classroom improvement strategy is generated based on the first dynamic scoring vector and the corresponding target score to guide adjustments to classroom teaching content. This approach achieves real-time perception and dynamic scoring of the classroom teaching process, resulting in more accurate classroom improvement strategies. Therefore, it improves the real-time nature, comprehensiveness, and guidance of course teaching evaluation, enabling real-time adjustments to teaching plans, optimization of teaching content, and enhancement of classroom teaching quality.

[0060] This application provides a dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking, such as... Figure 1As shown, the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking provided in this application includes the following steps:

[0061] S101, in response to the teaching optimization request, obtains the first multimodal data of the current classroom teaching process.

[0062] Specifically, step S101 can be implemented as follows: In response to the teaching optimization request, a preset cache detection module is activated; the cache detection module is used to query whether there is cached data for the teaching optimization request; if there is no cached data for the teaching optimization request, the multimodal data acquisition module is activated to obtain the first multimodal data of the current classroom teaching process; if there is cached data for the teaching optimization request, the preset improvement strategy in the cached data is set as the classroom improvement strategy of the current classroom and published.

[0063] In this application embodiment, different business scenarios are pre-set, including at least: fixed feedback scenario, local feedback scenario, or cloud scenario.

[0064] Fixed feedback scenario: refers to a predefined fixed time point (such as every 10 minutes after the start of the course) where teaching optimization requests are automatically triggered without the need for administrators or teachers to initiate them.

[0065] Local feedback scenario: refers to teaching optimization requests initiated by teachers or administrators in the classroom through terminal devices or mobile devices, with the local server completing data collection and broadcasting of optimization strategies;

[0066] Cloud-based scenario: This refers to a scenario where data is uploaded from the classroom to a cloud platform for processing, the cloud server generates classroom improvement strategies, and the results are sent to local terminals. This is typically used in remote classrooms or resource-constrained environments.

[0067] It is understood that, based on the different business scenarios in the embodiments of this application, the above-mentioned teaching optimization request can be automatically triggered or passively triggered.

[0068] In this embodiment, when a processing device (e.g., a local server, cloud server, edge computing box, etc.) receives a teaching optimization request, it first activates a preset cache detection module in response to the request. The cache detection module queries whether cached data generated by the teaching optimization request exists. This cached data includes at least dynamic evaluation vectors or classroom improvement strategies. If the cache detection module detects cached data for the teaching optimization request, it directly calls and publishes the classroom improvement strategies from the cached data, thereby reducing computational burden and improving response speed.

[0069] The aforementioned teaching optimization request may include at least one of the following query identifiers: Teacher ID, Course ID, and Class Hour ID. It is understood that the teaching optimization request in this embodiment can be composed of Teacher ID, Course ID, and Class Hour ID concatenated together; that is, the teaching optimization request is a query variable formed by concatenating Teacher ID, Course ID, and Class Hour ID.

[0070] Furthermore, in this embodiment, the optimized classroom teaching data is stored as cached data with a preset expiration time. This cached data may include key information such as teacher ID, course ID, lesson ID, classroom improvement strategy, and time window (voice fingerprints may also be added). Upon receiving a teaching optimization request, the cached data corresponding to the request is queried based on the query identifier in the request. If found, it is determined whether the Time To Live (TTL) value exceeds the preset expiration time. If the TTL exceeds the preset expiration time, it is determined that no cached data for the teaching optimization request exists; otherwise, it is determined that cached data for the teaching optimization request exists.

[0071] For example, if a teacher has previously requested a classroom improvement strategy for this lesson, the system first constructs a unique cache key based on key information such as teacher ID, course ID, lesson ID, and time window (voice fingerprints can also be added). It then queries the cache layer for this entry, simultaneously verifying whether the Time To Live (TTL) value has expired (or whether the optional audio fingerprint matches). If the query finds a match and the verification passes, it is determined to be a "valid cache," meaning that cached data for the teaching optimization request exists. In this case, the preset improvement strategy in the cached data is directly used as the classroom improvement strategy for the teaching optimization request, without the need for re-collection and analysis.

[0072] In this embodiment, if no cached data corresponding to the teaching optimization request exists, the system determines that "cache status identifier = none" and transmits this identifier along with the teaching optimization request to the downstream module, initiating the real-time data collection, scoring, and teaching optimization process. Initiating the real-time data collection and scoring process ensures that the system has both high response efficiency and supports personalized processing and dynamic calculation.

[0073] If no cached data is detected, a "cache status identifier" and a "business scenario identifier" are generated as the basis for subsequent data processing.

[0074] The classroom improvement strategies obtained in this application embodiment can be announced via voice or text, and no specific limitation is made in this application embodiment.

[0075] Furthermore, to ensure the smooth progress of classroom teaching and avoid interruptions, after obtaining a classroom improvement strategy using the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking provided in this application embodiment, prompt information can be generated according to the classroom improvement strategy and sent to the teacher's terminal device (e.g., teacher's all-in-one machine, teacher's mobile phone) in the form of text or vibration, so as to avoid directly affecting teaching through voice broadcasting. Additionally, if the user needs to announce via voice, the announcement can be made only after receiving the broadcast instruction.

[0076] In this embodiment, the multimodal data acquisition module may include multimodal acquisition devices such as high-definition cameras, microphone arrays, and smart interactive terminals, for collecting voice, images, and interaction logs during classroom teaching.

[0077] In some embodiments of this application, the first multimodal data includes at least one of text data, sentiment score vector, and video feature vector, as well as event timestamp and spatial location information in the current classroom.

[0078] In this embodiment, the spatial location information in the current classroom is acquired through acquisition devices such as high-definition cameras. The spatial location information referred to here means the relative position information (e.g., three-dimensional coordinates) of the image acquisition device (e.g., a camera) or the indirectly associated spatial attributes of area labels (e.g., teacher area, student area). It is understood that voice data, video data, and interaction logs may contain relevant spatial location information; therefore, voice data, video data, and interaction logs can be identified to obtain richer spatial location information.

[0079] Specifically, in this embodiment of the application, the acquisition of the first multimodal data of the current classroom teaching process through the multimodal data acquisition module can be achieved in the following ways:

[0080] The multimodal data acquisition module collects classroom activity data for the current class. This data includes at least: audio data, video data, interaction logs, and spatial location information within the classroom. The audio data is then converted into text data. Based on the acoustic parameters of the text and audio data, an emotion score vector is obtained using a pre-defined text emotion model. Facial expression features and scene features are extracted from each frame of the video data, and a video feature vector is generated using these features. The interaction logs are analyzed to determine the valid events within each time period, and the time of each valid event is set as an event timestamp.

[0081] In the absence of cached data, this embodiment of the application activates the multimodal data acquisition module, which uses multimodal acquisition devices (microphone arrays, high-definition cameras, and smart interactive terminals, etc.) deployed in the classroom to collect and process voice data, video data, and interaction logs during classroom teaching in real time.

[0082] As can be seen from the above, this application requires voice data processing, video data processing, and interactive log processing to obtain the first multimodal data.

[0083] Specifically, the above-mentioned voice data processing is as follows:

[0084] Noise suppression is performed on the acquired raw speech signal, preferably using Wiener filtering or spectral subtraction to enhance speech clarity. Then, an Automatic Speech Recognition (ASR) model based on the Transformer architecture is used to convert the speech into structured text T, i.e., the aforementioned text data.

[0085] For the sentiment analysis stage, this application introduces a dual-channel feature fusion mechanism. Firstly, acoustic parameters (such as speech rate, pitch, and volume curve) are extracted from the speech data. Then, sentiment classification is performed using the acoustic parameters and the text obtained from speech conversion. A BERT+BiLSTM text sentiment recognition model is used to output a sentiment score vector. The sentiment score vector covers sentiment categories (such as positive, neutral, negative) and sentiment intensity scores (range [0,1]).

[0086] Specifically, the video data processing described above is as follows:

[0087] The acquired video data can be parsed by the CNN-Transformer hybrid architecture image processing module to extract facial expression features from each frame of the video data, including the facial expressions of teachers and students. Then, the MediaPipe or OpenFace model is used to identify 68 facial key points and perform emotion classification. Simultaneously, scene features (such as raising hands, standing, and group gatherings in discussion groups) are extracted using object detection algorithms such as YOLOv5, and then a video feature vector is generated. Used for the construction of subsequent spatiotemporal semantic graphs. In this embodiment, all video feature vectors are organized in frame-level temporal order and synchronized with the audio timeline.

[0088] Specifically, the interaction log processing is as follows:

[0089] The interaction logs (including valid events such as text comments, votes, and homework submissions) from teaching platforms or terminals are standardized in format. Each record is marked with a timestamp, the interaction type and content are extracted, and they are encoded into a structured record with a unified format for easy integration with voice and video data. In this embodiment, the first multimodal data is:

[0090] ;

[0091] in, Text data; This is an emotion score vector used to represent tone, speech rate, emotion category, and intensity; This is the event timestamp, with precision in seconds; This refers to spatial location information, specifically the relative position information (e.g., three-dimensional coordinates) of image acquisition devices (e.g., cameras) or area labels (e.g., teacher area, student area); Video feature vectors, which integrate facial expression recognition and motion detection information.

[0092] In this embodiment of the application, the above data format can be used as the unified format of the first multimodal data to obtain standardized multimodal data, providing a high-quality structured foundation for the subsequent construction of spatiotemporal semantic graphs.

[0093] S102, map each piece of first multimodal data to a node in the first spatiotemporal semantic graph, and generate the first spatiotemporal semantic graph according to the preset edge construction conditions and preset edge generation conditions.

[0094] Specifically, each piece of first multimodal data can be mapped to a node in the first spatiotemporal semantic graph. Then, based on the preset edge construction conditions, the time difference, spatial Euclidean distance, and semantic similarity between the two nodes are calculated respectively. Based on the time difference, spatial Euclidean distance, and semantic similarity, directed edges of the first spatiotemporal semantic graph are constructed between the two nodes that meet at least two preset edge generation conditions, thereby obtaining the first semantic graph.

[0095] In this embodiment of the application, each piece of first multimodal data in classroom teaching activities is mapped to a node of a first spatiotemporal semantic graph. Each node contains the following attributes:

[0096] ;

[0097] Text data; This is an emotion score vector used to represent tone, speech rate, emotion category, and intensity; This is the event timestamp, with precision in seconds; This refers to spatial location information, specifically the relative position information (e.g., three-dimensional coordinates) of image acquisition devices (e.g., cameras) or area labels (e.g., teacher area, student area); It is a video feature vector that integrates facial expression recognition and motion detection information.

[0098] In this embodiment, valid events within each time period generate corresponding node sets. .

[0099] The preset edge construction conditions in this application embodiment include time constraints, spatial constraints, and semantic similarity constraints.

[0100] The above time constraint: if two nodes , Time difference Then the time constraint condition is satisfied, where, To preset the time difference threshold, The value range can be set to 5~10 seconds to capture continuous interactions within a short period of time.

[0101] The above spatial constraints: calculate two nodes , Spatial Euclidean distance ,like This is considered to satisfy the time constraint within adjacent spatial regions. (Preset distance threshold) It can be configured according to the classroom structure, and a common setting is 1.5 meters (about the distance between two desks and seats).

[0102] The above semantic similarity constraint: calculate the semantic similarity between two nodes. , Text data in , Embedded into vectors, for example, using a BERT model. , And calculate the cosine similarity (i.e., semantic similarity):

[0103] ;

[0104] like If they are semantically related, then a pre-defined similarity is assumed. It can be set to 0.7~0.8 to connect statements that have a logical continuity relationship.

[0105] The above-mentioned preset edge generation conditions: when node pairs If at least two of the above preset edge generation conditions are met, an edge from... point to Directed edge .

[0106] Through the above embodiments, the node set and edge set By combining these elements, this application forms a complete spatiotemporal semantic graph: .

[0107] Furthermore, after step S102, the embodiments of this application may further include the following method: determining the teaching scenario in the classroom teaching process, wherein the teaching scenario includes at least one of lecturing, answering questions and discussion; then clustering the first spatiotemporal semantic graph according to the teaching scenario to obtain a semantic subgraph of each teaching scenario, and setting each semantic subgraph as the first spatiotemporal semantic graph.

[0108] In this embodiment of the application, in order to enhance the modeling capability of different teaching segments (such as lecturing, Q&A, and discussion), the first spatiotemporal semantic graph can be modeled based on the semantic features and interaction density of nodes. The division is as follows:

[0109] (1) Use an embedding-based clustering partitioning method;

[0110] (2) Each semantic subgraph represents a relatively independent teaching scenario (such as “group discussion 1” or “question and interaction paragraph”);

[0111] (3) Each semantic subgraph retains its original edge structure and serves as the input unit for the subsequent scoring stage.

[0112] The first spatiotemporal semantic graph and its semantic subgraphs not only preserve the temporal and spatial information of events, but also integrate semantic and emotional attributes, providing structured input for subsequent graph neural networks and supporting in-depth classroom behavior modeling and state assessment.

[0113] "Topic Subgraph Division" automatically breaks down the complex spatiotemporal semantic graph of an entire lesson into several semantically concentrated and temporally continuous segments of uncontrolled semantic graphs. This allows for more precise analysis of the performance of each teaching segment, supporting more granular and personalized scoring and suggestions, thereby further improving teaching quality.

[0114] S103, input the first spatiotemporal semantic graph into the preset graph neural network to obtain the first dynamic scoring vector in the course teaching process.

[0115] In this embodiment of the application, the first spatiotemporal semantic graph constructed in step S102 above is used. The input is a pre-defined Graph Neural Network (GNN), which uses a Graph Convolutional Network (GCN) to extract features and fuse information from the first spatiotemporal semantic graph, ultimately generating a first dynamic scoring vector to represent the classroom state, as shown below:

[0116] 1. Initial feature vector embedding, as follows:

[0117] For each node in the first spatiotemporal semantic graph First, its attributes Perform unified embedding processing:

[0118] 1) Text data Semantic vectors are extracted using the BERT model;

[0119] 2) Emotional Score Vector With video feature vectors Directly spliced ​​and then projected;

[0120] 3) Event timestamp and spatial location information Encoded as sine and cosine time features and spatial location vectors;

[0121] Finally, the initial node vector is generated. Its dimensions are usually set to 128~256, which are used as inputs to GCN.

[0122] in, The specific operations include: converting text data Transformed into a 768-dimensional vector using the BERT model; sentiment score vector (3D) Video Feature Vector (128 dimensions), timestamp t (encoded as a 2-dimensional sine and cosine vector), and spatial location s (encoded as a 3-dimensional coordinate vector) are concatenated to form a vector with a total dimension of 768(T) + 3( )+128 The original feature vector of 904 dimensions (t + 2(s) + 3(s)) is projected to 256 dimensions through a fully connected layer. .

[0123] 2. The structure and parameter settings of the graph convolutional network are as follows:

[0124] The graph neural network in this embodiment can adopt a three-layer graph convolutional structure (3-layer GCN) with L=3 layers, which is used to capture the multi-order adjacency information of nodes in the spatiotemporal semantic graph.

[0125] The feature update formula for each layer is:

[0126] ;

[0127] in: The feature representation of node v at layer l; Let v be the set of adjacent nodes; , Here are the weight matrix and bias terms for the l-th layer; For the activation function, the ReLU (Rectified Linear Unit) activation function can be selected in the embodiments of this application, which has both non-linear expressive power and training stability.

[0128] In this embodiment, the node vector dimension of each layer can be set as follows: First layer: input dimension 128, output dimension 128; Second layer: input dimension 128, output dimension 64; Third layer: input dimension 64, output dimension 32.

[0129] In this embodiment, by using layer-by-layer dimensionality reduction compression, the high-order semantic structure features of nodes can be effectively extracted.

[0130] 3. Graph-level pooling and rating mapping, as detailed below:

[0131] After completing node-level feature learning, global feature aggregation is performed on the entire spatiotemporal semantic graph to generate a unified graph-level table. The calculation is performed using a global average pooling strategy as follows:

[0132] ;

[0133] in The calculation logic is as follows: For the final feature vector of all nodes... (L represents the number of GCN layers, e.g., 3 layers, output 32 dimensions) Take the average value, where This represents the total number of nodes. Example: If there are 10 nodes, the total number of nodes for each node is... If it is a 32-dimensional vector, then The average of the dimensions corresponding to the 10 vectors is still 32 dimensions.

[0134] This graph-level vector comprehensively reflects the multimodal state information throughout the entire classroom teaching process and has good discriminative ability.

[0135] In this embodiment of the application, an additional fully connected network (Dense Layer) is used to... Mapped to the first dynamic rating vector:

[0136] ;

[0137] in (or more dimensions), The weight matrix is ​​32×3 (32-dimensional). Mapped to a 3D rating vector. The bias term is a 3-dimensional term, which is obtained through supervised learning (using manually labeled ratings as labels).

[0138] Each component corresponds to the following evaluation dimensions: (1) Teacher's teaching quality score; (2) Student participation score; (3) Classroom atmosphere score.

[0139] In this embodiment, the first dynamic scoring vector can be stored in a local cache in real time for use in subsequent improvement suggestion generation modules, or it can be pushed to the teaching management platform via an interface.

[0140] The graph neural network module provided in this application embodiment can adopt a supervised learning mechanism, setting the training objective as minimizing the residual between the model scoring result and the real score or historical high-quality score. The real score can be labeled by teachers or administrators, or it can be provided by a predefined library of high-quality classroom case studies.

[0141] S104, based on the preset strategy network, and according to the first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector, generates classroom improvement strategies.

[0142] In this embodiment of the application, the policy network corresponding to the first spatiotemporal semantic graph can be obtained first; and the current classroom state can be set according to the deviation between the first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector; and the current classroom state can be input into the policy network corresponding to the first spatiotemporal semantic graph to generate a classroom improvement strategy.

[0143] As can be seen from the above embodiments, the first spatiotemporal semantic graph in the embodiments of this application can be a divided semantic subgraph. In this case, more targeted policy networks can be adopted respectively, thereby further improving the quality of classroom teaching optimization.

[0144] The first dynamic scoring vector is obtained through the above step S103. Then, it is compared with the target scoring vector in the historical high-quality classroom case study database. The two are compared, and then the current classroom state representation st is calculated based on the deviation between them. This st representation is then input into a pre-defined policy network to generate targeted classroom improvement strategies, as shown below:

[0145] 1. Modeling of state representation and scoring bias, as detailed below:

[0146] Classroom status Defined as the deviation between the first dynamic rating vector and the target rating vector:

[0147] ;

[0148] in, The problem stems from the teacher's poor teaching quality; For student participation deviation; The classroom atmosphere was off.

[0149] For example, if = [80, 75, 85] (Teaching quality, participation, atmosphere). = [90, 85, 90], then =[10,10,5].

[0150] Among them, the quality of teachers' teaching is calculated by weighting factors such as the completeness of teaching process (e.g., coverage of knowledge points), clarity of language expression (speech recognition fluency score), and logic of blackboard writing / PPT (text semantic coherence score). The weights can be set by experts (e.g., coverage accounts for 40%, clarity accounts for 30%, and logic accounts for 30%).

[0151] Student participation is quantified based on factors such as speaking frequency (number of interactions per 5 minutes), question quality (question relevance score to the course), and body activity (number of times hands are raised / nods during video recognition). For example, participation = (speaking frequency × 0.5 + question quality × 0.3 + body activity × 0.2).

[0152] Classroom atmosphere: Combines student emotional score vector (proportion of positive emotions), group interaction intensity (proportion of group discussion time), etc., such as atmosphere score = proportion of positive emotions × 0.6 + proportion of discussion time × 0.4.

[0153] Deviation calculation: The deviation for each dimension is the difference between the target value and the actual value, i.e. .

[0154] Regarding the teacher teaching quality evaluation method (used to calculate Δ_teach), the evaluation of teacher teaching quality requires first quantifying each sub-item (completeness of teaching process, clarity of language expression, and logicality of blackboard writing / PPT), and then obtaining the total score through weighting, as follows:

[0155] ① Completeness of teaching process (coverage of knowledge points)

[0156] Data sources: a list of knowledge points pre-set in the course syllabus, transcripts of teachers' lectures, or PPT content.

[0157] Scoring method: The ratio of the number of knowledge points actually covered by the teacher (denoted as K) to the total number of knowledge points required by the syllabus (denoted as K_total) is calculated as follows: Coverage score = (K / K_total) × 100. For example, if the syllabus requires 10 knowledge points and the teacher actually explains 8, the coverage score is (8 / 10) × 100 = 80.

[0158] ② Clarity of speech expression (fluency of speech recognition)

[0159] Data source: Teacher's voice signal collected by microphone, converted into text by ASR (Automatic Speech Recognition) model and recognition log.

[0160] Scoring Method: Calculated based on speech recognition accuracy and coherence: Clarity Score = (Accuracy × 0.7 + Coherence × 0.3) × 100, where Accuracy = number of correctly recognized words / total number of words (e.g., 95 out of 100 correctly recognized words have an accuracy of 0.95), and Coherence = percentage of sentences without pauses or stutters (e.g., 8 out of 10 sentences are fluent, with a coherence of 0.8). For example, with an accuracy of 0.95 and a coherence of 0.8, the Clarity Score = (0.95 × 0.7 + 0.8 × 0.3) × 100 = 90.5.

[0161] ③ The logical structure of the blackboard writing / PPT (the semantic coherence of the text)

[0162] Data source: OCR text of whiteboard photos, text content of PPT.

[0163] Scoring method: The semantic similarity of adjacent content is calculated using the Text Semantic Model (BERT), and the average value is taken: Logical score = Average semantic similarity × 100. For example, if the semantic similarities of three consecutive chapters in a PPT are 0.8, 0.9, and 0.7 respectively, then the average similarity is 0.8, and the logical score is 80.

[0164] ④ Total score for teacher's teaching quality

[0165] The teacher's teaching quality score is calculated based on the weights set by experts (e.g., coverage 40%, clarity 30%, logic 30%): Coverage score × 0.4 + Clarity score × 0.3 + Logic score × 0.3.

[0166] Regarding the student engagement scoring method (used to calculate Δ_engage), student engagement needs to be quantified and weighted separately for speaking frequency, question quality, and physical activity, as follows:

[0167] ① Speaking frequency (number of interactions per 5 minutes)

[0168] Data sources: ASR recognition results of students' speech and text recordings of classroom interactive terminals.

[0169] Scoring method: The total number of times students actively speak (including voice and text) within every 5 minutes is counted and mapped to a score of 0-100: Speaking frequency score = min(actual number of times / preset number of times to meet the standard × 100, 100). For example, if the preset number of times to meet the standard within 5 minutes is 5 times, and the actual number of times is 3, then the score = 3 / 5 × 100 = 60.

[0170] ② Question quality (relevance of questions to the course)

[0171] Data sources: Text content of student questions (voice transcription or text input), course knowledge point tags.

[0172] Scoring method: A text matching model (TFIDF or BERT) is used to calculate the relevance score between the question and the course knowledge point (range 0~1), which is then converted to a percentage: Question quality score = relevance score × 100. For example, if a student asks "How to use a quadratic function to find the maximum or minimum value?" and the relevance score to the course knowledge point "properties of quadratic functions" is 0.9, then the score = 90.

[0173] ③ Physical activity level (number of times hands are raised / heads are nodded)

[0174] Data source: Video stream from a camera inside the classroom, showing student body movements identified by a target detection model (YOLOv5).

[0175] Scoring method: The total number of times students raise their hands and nod within every 5 minutes is counted and mapped to a score of 0-100: Physical activity score = min(actual number of actions / preset number of times the target is met × 100, 100). For example, if the preset number of actions to meet the target is 10 within 5 minutes, and 6 actions are actually detected, then the score = 60.

[0176] ④ Total score for student participation

[0177] Calculated using weighted averages (frequency of speaking 50%, quality of questions 30%, physical activity 20%): Student participation score = Frequency of speaking score × 0.5 + Quality of questions score × 0.3 + Physical activity score × 0.2

[0178] The classroom atmosphere scoring method (used to calculate Δ_mood) is quantified based on student emotional scores and the intensity of group interaction, as detailed below:

[0179] ① Student Emotional Score Vector (Percentage of Positive Emotions)

[0180] Data source: Video recognition results of students' facial expressions (classified as positive, neutral, and negative using the OpenFace model).

[0181] Scoring method: Calculate the percentage of students with positive emotions out of the total number of students: Percentage of students with positive emotions = Number of students with positive emotions / Total number of students. For example, if 21 out of 30 students show positive emotions, then the percentage of students with positive emotions = 0.7.

[0182] ② Group interaction intensity (percentage of group discussion time)

[0183] Data source: Duration of group discussion scenes (such as multiple people gathering and physical interaction) detected in video streams.

[0184] Grading method: Calculate the proportion of total group discussion time to total class time: Discussion time percentage = Total discussion time / Total class time. For example, if 15 minutes of discussion time is in a 45-minute class, then the percentage = 15 / 45 ≈ 0.333.

[0185] ③Total score for classroom atmosphere

[0186] Calculated by weight (positive emotions 60%, discussion time 40%): Classroom atmosphere score = Positive emotions percentage × 0.6 × 100 + Discussion time percentage × 0.4 × 100. For example, if positive emotions account for 0.7 and discussion time accounts for 0.333, then the score = 0.7 × 60 + 0.333 × 40 ≈ 42 + 13.3 = 55.3.

[0187] The scores for each dimension are obtained through the process of "sub-item quantification → percentage conversion → weighted summation" to form the final Score_Vector (teaching quality, participation, atmosphere). The Score_Vector is then compared with the Score_Target to obtain the deviations Δ_teach, Δ_engage, and Δ_mood, which provide input for the policy network.

[0188] The aforementioned deviations reflect the current classroom's deficiencies in key assessment dimensions, serving as the input state for the pre-designed strategy network.

[0189] 2. Policy network structure and training method, as detailed below:

[0190] In this embodiment, the policy gradient method can be used for policy optimization, and the aforementioned policy network... Output in state Choose classroom improvement strategies The probability distribution. The parameter update formula is:

[0191] ;

[0192] in: For policy network parameters; The learning rate (e.g., 0.001); This is the actual reward value; This serves as a baseline value, used to reduce variance and improve stability. For the policy network in state Select action log probability of parameters The gradient.

[0193] 3. Reward function design, as follows:

[0194] In this embodiment of the application, the reward value The design is based on the improvement effect of teaching assessment, and mainly considers the new dynamic assessment vector after the recommendations are implemented. (i.e., the change in the second dynamic scoring vector) and historical deviation:

[0195] ;

[0196] in: This represents the original scoring bias; The scoring deviation is due to the implementation of the recommendations; To increase the weighting factor for the score, control the intensity of positive incentives; Cost factors (such as the system resources required for complex feedback); To generate classroom improvement strategies The system cost function (such as speech synthesis time or bandwidth usage);

[0197] This reward function can effectively measure the positive teaching effect of the suggestions while constraining resource consumption.

[0198] 4. Baseline value The setup method is as follows:

[0199] benchmark value The following methods can be used to dynamically set it:

[0200] (1) Moving average method: Take the average of recent (e.g., the past 20 times) reward values ​​and set it as the current benchmark. This is used to smooth out training fluctuations;

[0201] (2) Historical best value comparison method: set as the average reward after the recommended implementation under the same type of course.

[0202] The gradient estimation can be stabilized through the processing described in the above embodiments, thereby improving the policy convergence efficiency.

[0203] 5. Personalized suggestion generation strategy, as detailed below:

[0204] Based on the current business scenario, generate different types of improvement suggestions:

[0205] (1) Fixed feedback scenario: Directly retrieve suggested content that matches the deviation pattern from the pre-set template library, suitable for low-interaction devices or standard courses;

[0206] (2) Local feedback scenario: Combine template suggestions with user behavior variables (such as weak teacher tone and little student interaction) and generate custom voice broadcast content through local speech synthesis module;

[0207] (3) Cloud feedback scenario: The system calls the cloud big model service to generate suggested content and outputs speech through streaming synthesis technology (such as Tacotron2 + WaveGlow), while caching the results to optimize response speed.

[0208] The dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking provided in this application will ultimately produce classroom improvement strategies that will be broadcast to the terminal in the form of audio or text for teachers to use for classroom improvement.

[0209] In some embodiments of this application, after step S104, the application may further include the following steps: when the system detects that a teacher is implementing a classroom improvement strategy, it will respond to the improvement instruction based on the feedback of the classroom improvement strategy and initiate a new round of data collection; acquire second multimodal data of the classroom teaching process within a preset time period, wherein the second multimodal data includes at least one of: text data, emotion score vector, video feature vector, event timestamp, and spatial location information in the current classroom; map each piece of second multimodal data to a node in a second spatiotemporal semantic graph, and generate a second spatiotemporal semantic graph based on preset edge construction conditions and preset edge generation conditions; input the second spatiotemporal semantic graph into the graph neural network to generate a second dynamic scoring vector; update the strategy network according to the second dynamic scoring vector and the target score corresponding to the second dynamic scoring vector.

[0210] It is understood that the specific implementation methods for obtaining the second dynamic scoring vector in the above embodiments can all refer to the above steps S101 to S103, and will not be repeated here.

[0211] In this embodiment, after detecting the implementation of the classroom improvement strategy, an improvement instruction is generated. At this time, the second multimodal data of the classroom teaching process within a certain period of time (i.e., within a preset time period, such as 5 minutes) can be reacquired. Then, a second spatiotemporal semantic graph is constructed, and the dynamic scoring vector (i.e., the second dynamic evaluation vector) after the implementation of the classroom improvement strategy is obtained based on the second spatiotemporal semantic graph. The strategy network is updated based on the second dynamic scoring vector and the target score corresponding to the second dynamic scoring vector, thereby realizing small-batch self-supervised incremental training, real-time updating of the strategy network, and continuous optimization of model performance.

[0212] The target scores mentioned above are preset benchmark values, which are typical scoring data extracted from the historical high-quality classroom case study database, or manually preset standard values.

[0213] In this embodiment, a small-batch self-supervised online learning mechanism can be used to periodically perform lightweight incremental training on the graph neural network and policy network using new data. Specifically: update frequency: can be set to trigger online training once every 20 complete classroom records after suggestion execution; batch size: the default setting is 8-16 records to avoid model drift caused by large batches; learning rate control: a small learning rate is used during the online update phase. To reduce the risk of catastrophic forgetting; data buffering strategy: can retain recent... 50 high-quality labeled samples are used to form retraining samples to assist in training with a mixture of new and old data; Network freeze layer setting: During online training, the parameters of the first few layers in the graph neural network and policy network are frozen, and only the high-level feature fusion module and output layer are updated to improve learning efficiency.

[0214] In the update process of this application embodiment, the neural network parameters are iterated in the following ways: 1) The graph neural network optimizes the feature extraction accuracy by minimizing the score residual; 2) The policy network optimizes the policy by using the cumulative reward value as the objective function; 3) All update operations are completed asynchronously in the background without affecting the real-time feedback process at the front end.

[0215] In some embodiments of this application, a new dynamic scoring vector is generated based on the updated graph neural network. The system can display the following dynamic information through a graphical user interface (GUI): a rating indicator trend chart showing the curves of rating dimensions (teaching quality, participation, atmosphere, etc.) changing over time; a suggestion adoption effect chart showing the comparison of ratings before and after suggestion implementation, quantifying the improvement; a feedback response profile chart identifying the effectiveness of each suggestion, which suggestions have produced positive effects, and which have not yet met expectations, assisting teachers in optimizing teaching strategies; and model status prompts, including the current neural network version, the time of the last online update, and the cumulative number of learning rounds.

[0216] In this embodiment of the application, the user can select whether to mark the current data as high confidence feedback through the interface, and the system adjusts the priority of training data accordingly to strengthen the learning weight of the corresponding strategy.

[0217] The policy network provided in this application embodiment can employ unsupervised reinforcement learning based on reward feedback. Its reward signal is calculated based on the change in classroom scores after the implementation of the improved strategy. Specifically, the difference in score deviation before and after strategy implementation is used as a positive reward, and system resource consumption is combined as a negative weight to form an overall reward value. This reward signal is input into the policy network as training feedback to optimize the generated strategy. In other words, the reward signal guides the policy network to learn a "high score improvement - low resource consumption" generation strategy, achieving automation and personalization of teaching improvement.

[0218] For example, the classroom improvement strategy for the first lesson was "adding a 5-minute discussion session." After implementing the strategy, student interaction improved from 75 to 85, and the system recorded a reward of +10 (significantly effective). For the second lesson, the improvement strategy was "adding image displays." After implementation, the teaching quality improved from 88 to 89, a very small change, and the reward was +1 (minor effect). For the third lesson, the improvement strategy was "adding a 5-minute Q&A session." After implementing the strategy, the teaching quality improved from 75 to 73, and the reward was -2, indicating no effect.

[0219] like Figure 2 As shown, this application can first detect whether there is a valid cache (i.e., cached data corresponding to the teaching optimization request) through the cache detection module; if so, the classroom improvement strategy can be broadcast directly through the cached data; if not, the first multimodal data can be obtained through the multimodal data acquisition module, and then the first spatiotemporal semantic graph can be constructed based on the first multimodal data. Subsequently, the first spatiotemporal semantic graph is dynamically scored through the graph neural network to obtain the first dynamic scoring vector. Then, the classroom improvement strategy is generated and broadcast according to the strategy network. Subsequently, when it is detected that the teacher has executed the classroom improvement strategy, in response to the improvement instruction based on the feedback of the classroom improvement strategy, the second dynamic evaluation vector is obtained through the adaptive feedback mechanism. Then, the neural network parameters are updated and visualized through the second dynamic evaluation vector.

[0220] Understandably, the aforementioned improvement instructions are used to indicate that a teacher has been detected implementing classroom improvement strategies.

[0221] Compared with existing technologies, the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking provided in this application constructs a first spatiotemporal semantic graph reflecting the teaching process using first multimodal data during classroom teaching. Based on this technology, a graph neural network is used to perform data fusion and feature extraction on the first spatiotemporal semantic graph to obtain a first dynamic scoring vector during classroom teaching. Then, through a pre-set strategy network, a corresponding classroom improvement strategy is generated based on the first dynamic scoring vector and the corresponding target score to guide adjustments to classroom teaching content. This approach achieves real-time perception and dynamic scoring of the classroom teaching process, resulting in more accurate classroom improvement strategies. Therefore, it improves the real-time nature, comprehensiveness, and guidance of course teaching evaluation, enabling real-time adjustments to teaching plans, optimization of teaching content, and enhancement of classroom teaching quality.

[0222] Based on the above embodiments, this application also provides a smart terminal, the principle block diagram of which can be as follows: Figure 3 As shown. The aforementioned intelligent terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps of any of the aforementioned dynamic classroom optimization methods based on real-time spatiotemporal semantic graph tracing. The display screen of the intelligent terminal can be a liquid crystal display (LCD) or an e-ink display.

[0223] Those skilled in the art will understand that Figure 3 The block diagram shown is only a partial structural diagram related to the solution of this application and does not constitute a limitation on the smart terminal on which the solution of this application is applied. The specific smart terminal may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0224] In one embodiment, a smart terminal is provided, the smart terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of any of the dynamic classroom optimization methods based on real-time spatiotemporal semantic graph tracing provided in the embodiments of this application.

[0225] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the dynamic classroom optimization methods based on real-time spatiotemporal semantic graph tracing provided in this application.

[0226] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0227] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0228] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0229] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0230] In the embodiments provided in this application, it should be understood that the disclosed systems / terminal devices and methods can be implemented in other ways. For example, the system / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0231] If the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, electrical signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0232] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions are not in essence a departure from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A dynamic classroom optimization method based on real-time spatio-temporal semantic graph tracking, characterized in that, The method includes: In response to the teaching optimization request, acquire the first multimodal data of the current classroom teaching process; Each piece of the first multimodal data is mapped to a node in the first spatiotemporal semantic graph, and the first spatiotemporal semantic graph is generated based on preset edge construction conditions and preset edge generation conditions; The first spatiotemporal semantic graph is input into a preset graph neural network to obtain the first dynamic scoring vector in the classroom teaching process; Based on a preset strategy network, and according to the first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector, a classroom improvement strategy is generated. The method of generating classroom improvement strategies based on a preset strategy network and according to the first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector includes: Obtain the policy network corresponding to the first spatiotemporal semantic graph; and Calculate the deviation between the first dynamic scoring vector and the target score corresponding to the first dynamic scoring vector, and set it as the current classroom state. Specifically: ； in, Given the current classroom situation, This is the first dynamic scoring vector. This is the target scoring vector in the database of high-quality history lessons. For the deviation in the quality of teachers' teaching, For student participation deviation, The classroom atmosphere was off; The current classroom state is input into the policy network corresponding to the first spatiotemporal semantic graph to generate the classroom improvement strategy; In response to an improvement instruction based on feedback from the classroom improvement strategy, second multimodal data of the classroom teaching process within a preset time period is acquired; wherein, the improvement instruction is used to indicate that the teacher has been detected to be implementing the classroom improvement strategy; Each piece of the second multimodal data is mapped to a node in the second spatiotemporal semantic graph, and the second spatiotemporal semantic graph is generated based on preset edge construction conditions and preset edge generation conditions; The second spatiotemporal semantic graph is input into the graph neural network to generate a second dynamic scoring vector; The strategy network is updated based on the second dynamic scoring vector and the target score corresponding to the second dynamic scoring vector; The method further includes: performing policy optimization using a policy gradient method, wherein the policy network... Output the selected classroom improvement strategy under the current classroom state. The probability distribution and parameter update formula are as follows: ; in, For policy network parameters, For learning rate, This is the actual reward value. As the baseline value, The policy network selects an action in the current classroom state. log probability of parameters The gradient; The method also includes setting a reward function, which is used to measure the positive teaching effect of the suggestions, specifically: ； in, As a reward value, This represents the original scoring bias. To account for the scoring deviation after the recommendation is implemented, To increase the weighting factor for the score and control the intensity of positive incentives, As a cost factor, To generate classroom improvement strategies The system cost function.

2. The dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking according to claim 1, characterized in that, The process of responding to a teaching optimization request and acquiring the first multimodal data of the current classroom teaching process includes: In response to the teaching optimization request, the preset cache detection module is activated; The cache detection module is used to check whether there is cached data for the teaching optimization request. If the cached data for the teaching optimization request does not exist, the multimodal data acquisition module is activated to obtain the first multimodal data of the current classroom during the classroom teaching process.

3. The dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking according to claim 2, characterized in that, The method further includes: If the cached data for the teaching optimization request exists, the pre-improvement strategy in the cached data is set as the classroom improvement strategy for the current classroom and published.

4. The dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking according to claim 2, characterized in that, The first multimodal data includes at least one of the following: text data, sentiment score vector, video feature vector, event timestamp, and spatial location information in the current classroom; The step of acquiring the first multimodal data of the current classroom during the classroom teaching process through the multimodal data acquisition module includes: The multimodal data acquisition module collects classroom activity data for the current classroom, including: voice data, video data, interaction logs, and spatial location information within the current classroom; wherein, the spatial location information is obtained from the image acquisition device in the multimodal acquisition module. The speech data is converted into text to obtain the text data; The emotion score vector is obtained by using a pre-defined text emotion model based on the acoustic parameters of the text data and the speech data; and Extract facial expression features and scene features from each frame of the video data, and generate a video feature vector using the facial expression features and scene features; and The interaction logs are analyzed to determine the valid events within each time period, and the time of each valid event is set as the event timestamp.

5. The dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking according to claim 1, characterized in that, The step of mapping each piece of the first multimodal data to a node in the first spatiotemporal semantic graph, and generating the first spatiotemporal semantic graph based on preset edge construction conditions and preset edge generation conditions, includes: Each piece of the first multimodal data is mapped to a node in the first spatiotemporal semantic graph; Based on the preset edge construction conditions, calculate the time difference, spatial Euclidean distance, and semantic similarity between the two nodes respectively; Based on the time difference, the spatial Euclidean distance, and the semantic similarity, directed edges are constructed on two nodes that meet at least two of the preset edge generation conditions to form the first spatiotemporal semantic graph, so as to obtain the first spatiotemporal semantic graph. The preset edge generation conditions include: the time difference is less than or equal to a preset time difference threshold, the spatial Euclidean distance is less than or equal to a preset distance threshold, and the semantic similarity is greater than or equal to a preset similarity.

6. The dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracking according to claim 5, characterized in that, After constructing directed edges for the first spatiotemporal semantic graph from two nodes that meet at least two preset edge generation conditions based on the time difference, the spatial Euclidean distance, and the semantic similarity, the method further includes: The teaching scenarios in the classroom teaching process are determined, and the teaching scenarios include at least one of lecturing, Q&A, and discussion; The first spatiotemporal semantic graph is clustered and divided according to the teaching scenario to obtain a semantic subgraph for each teaching scenario, and each semantic subgraph is set as the first spatiotemporal semantic graph.

7. A smart terminal, characterized in that, The smart terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracing as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the dynamic classroom optimization method based on real-time spatiotemporal semantic graph tracing as described in any one of claims 1 to 6.

Images