Classroom interaction labeling system and method based on time point jump and electronic equipment

Abstract

The embodiment of the invention discloses a classroom interaction labeling system and method based on time point jump and electronic equipment, and belongs to the technical field of electric digital data processing. The classroom interaction labeling system based on time point jump comprises an audio and video data synchronous acquisition and alignment module, a label generation module, a cross-modal feature fusion and label representation module, a multi-dimensional label quantification and jump module and a self-adaptive visualization and interaction jump interface module. And the classroom interaction analysis precision and efficiency are effectively improved.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing technology, and more specifically, to a classroom interactive annotation system, method, and electronic device based on time point jumps. Background Technology

[0002] With the widespread application of online education and smart classrooms, the demand for refined analysis and efficient review of the teaching process is increasing. Classroom interaction, as an important basis for evaluating teaching effectiveness and teacher professional development, directly impacts the depth of educational research and the efficiency of teaching improvement through its recording, annotation, and retrieval capabilities. Current mainstream classroom video playback systems have significant limitations in supporting interactive analysis: First, time positioning relies on manually dragging a progress bar, lacking a navigation mechanism directly linked to interactive events, making it difficult for users to quickly jump to key teaching nodes such as questions, feedback, or discussions. Second, existing annotation functions generally use generic timestamps or simple text notes, failing to provide structured classification of classroom interactive behaviors and unable to distinguish between interaction types with different teaching intentions, such as "guided questioning" and "direct correction." Third, the coverage of interaction dimensions is insufficient, neglecting core interactive strategies recognized in educational linguistics, such as "restation," "expansion," "request for clarification," and "meta-language feedback," resulting in annotation results lacking theoretical support and analytical value. Finally, there is a lack of dynamic linkage between tags and audio / video playback; users cannot directly trigger time jumps by clicking semantic tags, severely restricting the efficiency of teaching reflection and case review. The aforementioned problems collectively make it difficult for existing technologies to support the needs of high-precision, multi-dimensional, and interactive classroom interaction analysis. There is an urgent need for a classroom interaction annotation and navigation system with a tag generation module at its core, which integrates interaction type semantics and timeline synchronization mechanism, in order to achieve the organic unity of structured annotation, intelligent tag generation, and one-click time point navigation. Summary of the Invention

[0003] This application provides a classroom interactive annotation system, method, and electronic device based on time-point jumps to solve the problems existing in the prior art.

[0004] According to a first aspect of the embodiments of this application, a classroom interactive annotation system based on time-point jump is provided, including an audio and video data synchronous acquisition and alignment module, a tag generation module, a cross-modal feature fusion and tag representation module, a multi-dimensional tag quantization and jump module, and an adaptive visualization and interactive jump interface module.

[0005] This solution enables the system to synchronously collect and align audio and video data from classroom recordings, extracting a global master clock reference and basic audio and video feature data streams, providing accurate time references for subsequent annotation and analysis. The tag generation module generates structured interactive tag sequences based on a preset rule base, effectively distinguishing different types of classroom interactions. The cross-modal feature fusion and tag representation module unifies the modeling of data from multiple modalities, enhancing the comprehensiveness and accuracy of the annotation results. The multi-dimensional tag quantification and jump module further improves the precision of annotation through classification quantification, ensuring that the annotation results are highly consistent with the teaching scenario. The adaptive visualization and interactive jump interface module enables dynamic linkage between the annotated data and the video player's timeline, allowing users to directly trigger time jumps by clicking tags, significantly improving the efficiency of teaching reflection and case review.

[0006] In one possible implementation, the audio and video data synchronization acquisition and alignment module includes a media stream demultiplexing unit, a master time base extraction unit, and an audio feature extraction submodule, wherein: The media stream demultiplexing unit is used to separate video data streams and audio data streams based on the basic characteristics of audio and video data streams; The master time reference extraction unit is used to extract the timestamp sequence as the global master clock reference of the system; The audio feature extraction submodule is used to convert classroom video files into timestamped text sequences and interactive tone parameter sequences.

[0007] This solution enables the media stream demultiplexing unit to separate independent video and audio data streams by parsing classroom recording files, providing a clear hierarchical structure for subsequent data processing. The master time reference extraction unit extracts timestamp sequences from container metadata or data packet header information, ensuring the accuracy of the global master clock reference. The audio feature extraction submodule employs deep learning algorithms to transform audio data into timestamped text sequences and interactive tone parameter sequences, providing rich input data for the label generation module. This process not only improves data processing efficiency but also lays a solid foundation for subsequent annotation and analysis.

[0008] In one possible implementation, the media stream demultiplexing unit supports a standard digital container format, and the master time base extraction unit extracts a sequence of timestamps from the container's metadata or packet header information.

[0009] This solution enables the media stream demultiplexing unit to support standard digital container formats such as MP4 and AVI, ensuring good system compatibility and the ability to process classroom recording files from different recording devices. The master time base extraction unit accurately extracts the timestamp sequence from the container's metadata or data packet header information, serving as the system's global master clock reference. This ensures time synchronization of all audio and video data and tag sequences with millisecond-level precision, providing a reliable time base for the collaborative work of subsequent modules.

[0010] In one possible implementation, the audio feature extraction submodule employs a deep learning-based speech recognition and tone analysis algorithm model.

[0011] This solution employs a deep learning-based speech recognition model (such as an end-to-end connected temporal classification architecture) and a mood analysis model (such as acoustic feature extraction based on convolutional neural networks). This enables the high-precision conversion of audio frame sequences into timestamped text sequences and interactive mood parameter sequences containing multi-dimensional acoustic feature vectors. This not only significantly reduces the character error rate in speech recognition but also enriches the dimensions of interactive analysis, providing high-quality input data for subsequent label generation and cross-modal fusion.

[0012] In one possible implementation, the rule base of the tag generation module employs a hybrid mechanism based on semantic templates and keyword matching.

[0013] This solution enables the tag generation module to accurately identify and classify classroom interactive behaviors through a hybrid mechanism of semantic templates and keyword matching. Semantic templates define the contextual patterns of interactive behaviors, such as "guided questioning" or "direct correction," while the keyword matching library contains a set of trigger words for each interaction type, such as "why" and "please explain." This hybrid mechanism not only improves the accuracy of interactive tag generation but also ensures the theoretical support and analytical value of the tagging results.

[0014] In one possible implementation, the cross-modal feature fusion and label representation module employs an attention-based multimodal fusion network.

[0015] This solution enables the cross-modal feature fusion and label representation module to achieve unified modeling of text, audio, and video data through a multimodal fusion network. The text encoder uses a bidirectional long short-term memory network to process text sequences and capture contextual semantic information; the audio encoder uses a one-dimensional convolutional network to process interactive tone parameter sequences and extract emotional features from speech; and the video encoder uses a three-dimensional convolutional network to process video frame sequences and capture body language information from teachers and students. The attention-based multimodal fusion network dynamically adjusts the weights of each modality, ensuring the comprehensiveness and accuracy of the fused label vector sequences.

[0016] In one possible implementation, the multi-dimensional label quantization and jump module includes a label sequence organization unit and a jump instruction generation unit.

[0017] This solution ensures the consistency of the chronological order of label results by sorting the label data. The jump instruction generation unit converts the sorted label sequence into a clickable list of elements, providing technical support for the adaptive visualization and interactive jump interface module. This process not only improves the usability of label data but also provides users with an intuitive operating experience, supporting quick location and jump to key teaching points.

[0018] In one possible implementation, the adaptive visualization and interactive navigation interface module integrates a video player, a tag list area, and a timeline visualization area.

[0019] This solution integrates a video player, a tag list area, and a timeline visualization area, enabling dynamic linkage between tag data and the video player's timeline. The video player supports multiple encoding formats, ensuring compatibility; the tag list area displays a list of clickable tags, allowing users to directly jump to specific time points by clicking on tags; and the timeline visualization area visually displays the distribution of tag data by overlaying a tag distribution chart. This design significantly improves user efficiency and experience.

[0020] According to a second aspect of the embodiments of this application, a classroom interaction annotation method based on time-point jump is provided, comprising the following steps: receiving a classroom recording file, and parsing and decoding it through an audio and video data synchronization acquisition and alignment module to extract audio and video basic feature data streams and text sequences that are precisely aligned with a global master clock reference; converting the text sequence into a structured interactive tag sequence aligned with a global master clock reference according to preset rules; fusing the audio and video basic feature data streams to generate a unified fused tag vector sequence; classifying and quantizing the interactive tag sequence, and outputting the final tag sequence and jump signal sequence; converting the tag sequence into a clickable list through an adaptive visualization and interactive jump interface module, and responding to the user's click operation to achieve precise jump to the audio and video playback point.

[0021] The above approach extracts basic feature streams and text sequences by parsing and decoding classroom video files, providing data support for subsequent operations. Processing the text sequences generates interactive tag sequences, ensuring semantic consistency in the annotation results. Fusion of the basic feature streams generates a unified fusion vector sequence, improving the comprehensiveness of the annotation results. Classification of the interactive tag sequences outputs the final tag sequence and jump signal sequence, enabling users to quickly locate key teaching points. By converting the tag sequences into a clickable list, precise jumps to audio and video playback points are achieved in response to user clicks. This series of steps working together effectively improves the accuracy and efficiency of classroom interaction analysis.

[0022] According to a third aspect of the embodiments of this application, the present invention provides an electronic device, including: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the classroom interactive annotation method based on time jump as described in the first aspect or any corresponding embodiment.

[0023] According to a fourth aspect of the embodiments of this application, the embodiments of this specification provide a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the classroom interactive annotation method based on time point jump as described in any of the preceding claims.

[0024] According to a fifth aspect of the embodiments of this application, this specification provides a computer program product or computer program, the computer program product including a computer program stored in a computer-readable storage medium; a processor of the computer device reads the computer program from the computer-readable storage medium, and when the processor executes the computer program, it implements the classroom interactive annotation method based on time point jump as described in any of the preceding claims.

[0025] According to a sixth aspect of the embodiments of this application, the present specification provides a fabric processing device that employs a classroom interactive annotation method based on time point jumps as described in any of the first aspects, or has electronic equipment as described in the third aspect.

[0026] The technical effects achieved by the second to sixth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the classroom interaction annotation method based on time point jump provided in the embodiments of this application; Figure 2This is a schematic diagram illustrating the specific process of a classroom interaction annotation method based on time point jumps provided in an embodiment of this application; Figure 3 This is a schematic diagram of an interface provided in an embodiment of this application; Figure 4 This is another schematic diagram of the interface provided in the embodiment of this application; Figure 5 This is yet another interface schematic diagram provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be understood that "multiple" as mentioned herein refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., do not necessarily imply that they are different.

[0030] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0031] As mentioned in the background technology, with the widespread application of online education and smart classrooms, the demand for refined analysis and efficient retrospective analysis of the teaching process is increasing. Classroom interaction, as an important basis for evaluating teaching effectiveness and teachers' professional development, has recording, annotation, and retrieval capabilities that directly affect the depth of educational research and the efficiency of teaching improvement. Current mainstream classroom video playback systems have significant limitations in supporting interactive analysis: First, time positioning relies on manually dragging a progress bar, lacking a navigation mechanism directly linked to interactive events, making it difficult for users to quickly jump to key teaching nodes such as questions, feedback, or discussions; second, existing annotation functions generally use generic timestamps or simple text notes, failing to provide structured classification of classroom interactive behaviors and unable to distinguish between interactive types with different teaching intentions, such as "guided questioning" and "direct correction"; third, the interactive dimensions are insufficiently covered, neglecting core interactive strategies recognized in educational linguistics, such as "restation," "expansion," "request for clarification," and "meta-language feedback," resulting in annotation results lacking theoretical support and analytical value; finally, there is a lack of dynamic linkage between tags and audio / video playback, preventing users from directly triggering time jumps by clicking semantic tags, severely restricting the efficiency of teaching reflection and case review. The aforementioned problems collectively make it difficult for existing technologies to support the needs of high-precision, multi-dimensional, and interactive classroom interaction analysis. There is an urgent need for a classroom interaction annotation and navigation system with a tag generation module at its core, which integrates interaction type semantics and timeline synchronization mechanism, in order to achieve the organic unity of structured annotation, intelligent tag generation, and one-click time point navigation.

[0032] This application provides a classroom interaction annotation system based on time-point jumps, such as... Figure 1 As shown, the system includes the following structure: The audio and video data synchronization acquisition and alignment module 101 is used to receive and parse classroom recording files. It separates video data streams and audio data streams through a media stream demultiplexing unit, and extracts the timestamp sequence as a global master clock reference through a master time reference extraction unit. At the same time, it uses an audio feature extraction submodule to convert the audio frame sequence into a text sequence with timestamps and an interactive tone parameter sequence, forming an audio and video basic feature data stream aligned with the global master clock reference.

[0033] The tag generation module 102, as the core of the system, has its input end connected to the output end of the text feature data stream. It is used to receive text sequences with timestamps and transform them into structured interactive tag sequences according to a preset rule base. The rule base defines multiple interaction type dimensions and their synonyms, including guided questioning, restatement, expansion, request for clarification, meta-language feedback, and direct correction. It also configures tag generation rules based on user input or automatic recognition for each dimension to generate tags composed of time points and specific words or their synonyms. The cross-modal feature fusion and label representation module 103 has its input end connected to the output end of the audio and video basic feature data stream. It is used to perform data fusion on the audio and video basic feature data stream and generate a unified fusion label vector sequence by combining text content, tone parameters and video features. This vector sequence is a precise digital representation of classroom interaction behavior at the corresponding time point.

[0034] The multi-dimensional tag quantization and jump module 104, whose input end is connected to the output end of the fused tag vector sequence and tag sequence, is used to classify and quantify interactive tags, evaluate the effectiveness of guided questions or the frequency of direct correction, and output the final tag sequence and jump signal sequence; the time-point jump engine, connected to the tag generation module 102, is used to organize the generated tags into a sequence according to time order, and generate a list of clickable tags in the user interface. When a tag is clicked, it automatically jumps to the corresponding audio and video playback point, supporting precise time-point navigation.

[0035] The adaptive visualization and interactive jump interface module 105 is connected to the time point jump engine and the tag generation module 102, and synchronized with the timeline of the video player. It is used to display the tag distribution map in real time on the user interface according to the tag sequence, and supports tag filtering, searching and multi-tag time point jump functions.

[0036] Preferably, the media stream demultiplexing unit in the audio and video data synchronization acquisition and alignment module 101 supports MP4 and AVI standard digital container formats. The main time reference extraction unit extracts a timestamp sequence from the container's metadata or data packet header information. This sequence serves as the system's global master clock reference, ensuring that the time of all subsequent processing flows is precisely aligned within milliseconds. The audio feature extraction submodule adopts a deep learning-based speech recognition and tone analysis algorithm model. The speech recognition model uses an end-to-end connected temporal classification architecture, and the tone analysis model extracts pitch, speech rate, and energy features based on a convolutional neural network. It converts the decoded audio frame sequence into a timestamped text sequence and an interactive tone parameter sequence. The character error rate of the text sequence is less than 5%, and the tone parameter sequence contains at least three dimensions of acoustic feature vectors.

[0037] Furthermore, the rule base of the tag generation module 102 adopts a hybrid mechanism based on semantic templates and keyword matching. The semantic template defines the contextual pattern of the interaction type, and the keyword matching library contains the trigger word set corresponding to each interaction type. The trigger word set for guided questions includes "Let's think about it" and "Think about it from another angle". The trigger word set for restatement includes "In other words" and "That is to say", the trigger word set for extension includes "In detail" and "Explain further", the trigger word set for requesting clarification includes "Please explain" and "What do you mean?", the trigger word set for meta-language feedback includes "Syntically" and "Word-wise", and the trigger word set for direct correction includes "Incorrect" and "The correct statement is". The rule base is also configured with a synonym mapping table to map user-input or automatically recognized terms to standard interaction types. When generating tags, the current timestamp is bound, and the tag format is "Interaction type at a specific time", with a time point precision of 0.1 seconds.

[0038] Furthermore, the cross-modal feature fusion and label representation module 103 adopts a multimodal fusion network based on an attention mechanism. This network includes a text encoder, an audio encoder, and a video encoder. The text encoder uses a bidirectional long short-term memory network to process text sequences, the audio encoder uses a one-dimensional convolutional network to process mood parameter sequences, and the video encoder uses a three-dimensional convolutional network to process video frame sequences. The fusion network calculates the weights between text, audio, and video features through a cross-modal attention layer to generate a unified fusion label vector sequence with a vector dimension of 128. Each vector corresponds to a multimodal interaction representation at a time point. The fused vector sequence is then reduced in dimension and normalized through a fully connected layer to output a standardized label vector with a dimension of 64.

[0039] Preferably, the multi-dimensional label quantization and jump module 104 employs a support vector machine-based classifier and a hidden Markov model-based sequence analyzer. The classifier scores the effectiveness of the interaction labels, with a score range of 0 to 1 and an effectiveness threshold set to 0.7. The sequence analyzer calculates the frequency distribution and transition probability of the interaction type and outputs the final label sequence and jump signal sequence. The jump signal sequence includes time point coordinates and interaction type identifiers. The time point coordinates are synchronized with the global master clock reference, and the interaction type identifiers are used to distinguish different categories of labels in the user interface. The model also integrates a real-time optimization unit that dynamically adjusts the classification threshold and sequence parameters based on user feedback. The optimization unit uses the gradient descent algorithm to update the model weights, with an update frequency of once every 10 minutes.

[0040] Furthermore, the time-based jump engine includes a tag sequence organization unit and a jump instruction generation unit. The tag sequence organization unit sorts the tags in chronological order, using a time sorting algorithm based on quicksort, with a time complexity of O(n log n). The jump instruction generation unit converts the sorted tag sequence into a list of clickable HTML elements, with each element bound to a JavaScript event listener. When a user clicks a tag, the event listener triggers a jump instruction containing a target time point parameter with a precision of 0.01 seconds. The jump engine also supports batch jump functionality, allowing users to select multiple tags, which the engine will jump to in chronological order. The jump interval is configurable, with a default interval of 2 seconds.

[0041] In addition, the adaptive visualization and interactive navigation interface module 105 integrates a video player, a tag list area, and a timeline visualization area. The video player supports H.264 and VP9 encoding formats. The tag list area displays a list of clickable tags, with the list items formatted as "hour, minute, second, interaction type". The timeline visualization area overlays a tag distribution map onto the video player's timeline, using different colors to mark different interaction types. The color mapping table predefines six colors corresponding to six interaction types. The visualization module also provides filtering and search interfaces. The filtering interface allows users to filter tags by interaction type, and the search interface supports keyword matching with a search response time of less than 100 milliseconds. The multi-tag time point navigation function is implemented through queue management, with a queue capacity of 50 tags.

[0042] like Figure 2 As shown, a classroom interaction annotation method based on time point jumps is described. The specific steps of this method are as follows: Step S201: Receive the classroom recording file and parse and decode it through the audio and video data synchronization acquisition and alignment module to extract the basic audio and video feature data stream and text sequence that are precisely aligned with the global master clock reference.

[0043] In this step, the classroom recording file is received and parsed and decoded through the audio and video data synchronization acquisition and alignment module. The basic audio and video feature data stream that is precisely aligned with the global master clock reference is extracted. The parsing process includes media stream demultiplexing, timestamp extraction, and audio feature conversion.

[0044] Step S202: Receive the text sequence and convert it into a structured interactive tag sequence aligned with the global master clock reference according to preset rules.

[0045] In this step, the text feature data stream is received through the tag generation module and transformed into a structured interactive tag sequence aligned with the global master clock reference according to preset rules. The transformation process includes semantic analysis, keyword matching, and synonym mapping.

[0046] Step S203: The audio and video basic feature data streams are fused to generate a unified fused label vector sequence.

[0047] In this step, the basic feature data streams of audio and video are fused through cross-modal feature fusion and label representation modules to generate a unified fused label vector sequence. The fusion process is implemented based on a multimodal network with an attention mechanism.

[0048] Step S204: Classify and quantize the interactive tag sequence, and output the final tag sequence and jump signal sequence.

[0049] In this step, interactive tags are classified and quantified through a multi-dimensional tag quantification and jump module, and the final tag sequence and jump signal sequence are output. The quantification process includes effectiveness scoring and frequency analysis.

[0050] Step S205: The label sequence is converted into a clickable list through the adaptive visualization and interactive jump interface module, and the precise jump to the audio and video playback point is realized in response to the user's click operation.

[0051] In this step, the tag sequence is transformed into a clickable list through the adaptive visualization and interactive jump interface module, and the precise jump to the audio and video playback point is realized according to the user's click operation. The jump process includes event listening and command sending.

[0052] For example, when the list has few contents, such as Figure 3 As shown, all clickable content can be displayed. The example in the image uses labels 1, 2, 3, and 4. When the list has a large amount of content, such as... Figure 4 As shown, you can display the main or frequently occurring content while hiding other content. When the corresponding display button is triggered, the hidden content will be revealed. In the image, labels 1, 2, and 3 represent the normally displayed content, and label 4 is the display button. Clicking label 4, as shown in the upper right corner of the image, reveals the hidden content. Simultaneously, label 4 changes from a display button to a hidden button.

[0053] In another example, when there are many content categories or many items belonging to the same button, the specific content is displayed in the form of a second-level menu, such as... Figure 5 As shown, when the original label 2 is clicked, the content of that label is expanded, as shown in the figure above as 21, 22, 23, and 24. Then, clicking the label under this secondary menu will jump to the next item.

[0054] The system and method described in this specification will be further explained below with reference to the embodiments.

[0055] Example 1 In primary and secondary school teaching settings, teachers use recording systems equipped with high-definition cameras and array microphones to record the entire classroom teaching process. (See also...) Figure 1 This system includes a synchronous audio and video data acquisition and alignment module, a tag generation module, a cross-modal feature fusion and tag representation module, a multi-dimensional tag quantization and jump module, a time-point jump engine, and an adaptive visualization and interactive jump interface module. The synchronous audio and video data acquisition and alignment module receives classroom recording files and separates the video and audio data streams through a media stream demultiplexing unit. This unit supports demultiplexing processing of MP4 and AVI standard digital container formats. The master time reference extraction unit extracts a timestamp sequence from the container's metadata or data packet header information as a global master clock reference. The timestamp accuracy reaches the millisecond level, ensuring accurate time alignment for all subsequent processing flows. The audio feature extraction submodule adopts a deep learning-based speech recognition and tone analysis algorithm model. The speech recognition model uses an end-to-end connected temporal classification architecture to convert audio frame sequences into timestamped text sequences, with a character error rate controlled within 5%. The tone analysis model extracts pitch, speech rate, and energy features based on a convolutional neural network, generating an interactive tone parameter sequence containing at least three dimensions of acoustic feature vectors, forming an audio and video basic feature data stream aligned with the global master clock reference.

[0056] The tag generation module, as the core of the system, connects its input to the output of the text feature data stream to receive timestamped text sequences. See also... Figure 2 The tag generation module transforms text sequences into structured interactive tag sequences based on a pre-defined rule base. The rule base employs a hybrid mechanism based on semantic templates and keyword matching. Semantic templates define the contextual patterns of interaction types, while the keyword matching library contains a set of trigger words for each interaction type. The trigger words for guided questions include phrases like "let's think about it" and "think about it from another angle"; the trigger words for restatements include expressions like "in other words" and "that is to say"; the trigger words for extensions include terms like "in detail" and "further explanation"; the trigger words for requests for clarification include questions like "please explain" and "what does it mean"; the trigger words for meta-language feedback include professional expressions like "grammatical" and "vocabulary"; and the trigger words for direct corrections include explicit instructions like "incorrect" and "the correct statement is correct." The rule base also includes a synonym mapping table, mapping user-inputted or automatically recognized terms to standard interaction types. Tags are generated by binding the current timestamp, and the tag format is a combination of a time point and the interaction type, with a time point precision of 0.1 seconds.

[0057] The input of the cross-modal feature fusion and label representation module is connected to the output of the audio / video basic feature data stream, and is used to perform data fusion on the audio / video basic feature data stream. See also Figure 3This module employs a multimodal fusion network based on an attention mechanism, comprising a text encoder, an audio encoder, and a video encoder. The text encoder uses a bidirectional long short-term memory network to process text sequences and extract semantic feature vectors; the audio encoder uses a one-dimensional convolutional network to process tone parameter sequences and extract acoustic feature vectors; and the video encoder uses a three-dimensional convolutional network to process video frame sequences and extract visual feature vectors. The fusion network calculates the weights between text, audio, and video features through a cross-modal attention layer, generating a unified fusion label vector sequence. The vector dimension is set to 128, with each vector corresponding to a multimodal interaction representation at a specific time point. The fused vector sequence undergoes dimensionality reduction and normalization through a fully connected layer, outputting a standardized label vector with a dimension of 64, forming a precise digital representation of classroom interaction behavior at the corresponding time point.

[0058] The multi-dimensional label quantization and jump module's input is connected to the output of the fused label vector sequence and label sequence, used for classifying and quantizing interactive labels. The model employs a support vector machine-based classifier and a hidden Markov model-based sequence analyzer. The classifier scores the effectiveness of interactive labels from 0 to 1, with a effectiveness threshold of 0.7; labels below this threshold are marked as invalid. The sequence analyzer calculates the frequency distribution and transition probability of interaction types, analyzing the temporal relationships between different interaction types. The model outputs the final label sequence and jump signal sequence. The jump signal sequence includes time point coordinates and interaction type identifiers. The time point coordinates are synchronized with the global master clock, and the interaction type identifiers are used to distinguish different categories of labels in the user interface. The model also integrates a real-time optimization unit that dynamically adjusts the classification threshold and sequence parameters based on user feedback. The optimization unit uses a gradient descent algorithm to update the model weights, with an update frequency of once every 10 minutes.

[0059] The time-based jump engine connects to the tag generation module to organize the generated tags into a sequence in chronological order. The tag sequence organization unit uses a time-based sorting algorithm based on quicksort, with a time complexity of O(n log n), ensuring efficient processing of large-scale tag data. The jump instruction generation unit converts the sorted tag sequence into a list of clickable HTML elements, with each element bound to a JavaScript event listener. When a user clicks a tag, the event listener triggers a jump instruction containing a target time point parameter with a precision of 0.01 seconds. The jump engine supports batch jump functionality; users can select multiple tags, and the engine jumps sequentially in chronological order. The jump interval is configurable, with a default interval of 2 seconds.

[0060] The adaptive visualization and interactive jump interface module 105 connects to the time-point jump engine and tag generation module, and is synchronized with the video player's timeline. This module integrates the video player, tag list area, and timeline visualization area. The video player supports decoding and playback of H.264 and VP9 encoding formats. The tag list area displays a list of clickable tags, with list items formatted as a combination of hours, minutes, seconds, and interaction type. The timeline visualization area overlays a tag distribution map onto the video player's timeline. The distribution map uses different colors to mark different interaction types. In one example, six tags are used as an illustration. The color mapping table predefines six colors corresponding to six interaction types. The visualization module provides filtering and search interfaces. The filtering interface allows users to filter tags by interaction type, and the search interface supports keyword matching, with a search response time controlled within 100 milliseconds. The multi-tag time-point jump function is implemented through queue management, with a queue capacity set to 50 tags to ensure stable performance when handling a large number of jump requests.

[0061] Example 2 In practical skills courses at vocational training institutions, instructors record demonstrations using mobile recording devices. The audio and video data synchronization acquisition and alignment module receives video files from mobile devices, while the media stream demultiplexing unit supports MOV and MKV format parsing, ensuring compatibility with video data from different sources. The master time base extraction unit adapts to the timestamp format specific to mobile devices, and the extracted timestamp sequence serves as the system's global master clock reference. The audio feature extraction submodule optimizes the noise suppression algorithm, filtering common background noise in the practical environment to improve speech recognition accuracy. The tag generation module expands the rule base, adding interactive types for practical guidance, including action demonstrations, key point emphasis, and error correction. The keyword matching library in the rule base is correspondingly expanded; the trigger words for action demonstrations include "Look at my action" and "Do it this way," the trigger words for key point emphasis include "Pay special attention" and "The key point is," and the trigger words for error correction include "You can't do it this way" and "The correct way is." The cross-modal feature fusion and label representation module, tailored to the characteristics of practical courses, strengthens the analysis weight of video features and extracts visual features such as gestures and tool usage through a 3D convolutional network. The multi-dimensional label quantification and jump module adjusts the classifier's threshold settings, lowering the validity threshold for practical category labels to 0.6, adapting to more lenient interactive judgment standards in skills teaching. The time-point jump engine optimizes batch jump logic, supporting grouped jumps by operation steps, with jump intervals adjustable according to operation complexity. The adaptive visualization and interactive jump interface module 105 enhances the timeline display function, supporting segmented marking of operation steps and key replays, improving the efficiency of skills learning.

[0062] The above describes the device embodiments of this application. For detailed descriptions of data, terms, nouns, specific execution processes of steps, technical problems and effects, alternative methods and combinations, please refer to the description in the method embodiments, which will not be repeated here.

[0063] This application also provides a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the time-point-based interactive classroom annotation method according to various embodiments of this specification as described in the "Exemplary Methods" section above.

[0064] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this specification. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages.

[0065] This application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor of the steps in the time-point jump-based classroom interactive annotation method according to various embodiments of this specification as described in the "Exemplary Methods" section above.

[0066] This application also provides an electronic device, including a memory and a processor. The memory stores a time-point-based classroom interaction annotation method, and the processor is used to employ the aforementioned time-point-based classroom interaction annotation method when executing the time-point-based classroom interaction annotation method.

[0067] Specifically, such as Figure 6 As shown, the electronic device includes a processor 100, at least one communication bus 200, a user interface 300, at least one external communication interface 400, and a memory 500. The communication bus 200 is configured to enable communication between these components. The user interface 300 may include a display screen, and the external communication interface 400 may include standard wired and wireless interfaces. The memory 500 stores a time-based interactive annotation method for classroom activities. The processor 100 is used to employ the aforementioned method when executing the time-based interactive annotation method stored in the memory 500.

[0068] The descriptions of the above computer program products, computer-readable storage media, and electronic devices are similar to those of the above method embodiments, and have similar beneficial effects. For any technical details not disclosed in the computer program products, computer-readable storage media, and electronic devices of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0069] The sequence numbers or order of description of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0071] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0072] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0073] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital versatile disc (DVD)), or a semiconductor medium (e.g., solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium. It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the scene data of the current frame in the 3D virtual scene involved in the embodiments of this application, the client's device information, and the scene interaction information are all obtained with full authorization.

[0074] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A classroom interaction annotation system based on time-point jumps, characterized in that, It includes a module for synchronous acquisition and alignment of audio and video data, a module for tag generation, a module for cross-modal feature fusion and tag representation, a module for multi-dimensional tag quantization and navigation, and a module for adaptive visualization and interactive navigation interface, among which: The audio and video data synchronization acquisition and alignment module is used to receive and parse classroom recording files, extract global master clock reference and audio and video basic feature data streams based on the classroom recording files, and generate a text sequence with timestamps based on the audio and video basic feature data streams. The tag generation module is used to receive the text sequence and generate a structured interactive tag sequence based on a pre-configured rule base and the text sequence. The cross-modal feature fusion and label representation module is used to combine the audio and video basic feature data stream and the text sequence to generate a unified fused label vector sequence. The multi-dimensional label quantization and jump module is used to classify and quantize the fused label vector sequence and the interactive label sequence to obtain the final label sequence and jump signal sequence. The adaptive visualization and interactive navigation interface module is used to convert the labeled data into a list of clickable tags synchronized with the video player's timeline and display the tag distribution map in the user interface.

2. The classroom interaction annotation system based on time-point jump according to claim 1, characterized in that, The audio and video data synchronization acquisition and alignment module includes a media stream demultiplexing unit, a master time base extraction unit, and an audio feature extraction submodule, wherein: The media stream demultiplexing unit is used to separate video data stream and audio data stream based on the audio and video basic feature data stream; The master time reference extraction unit is used to extract the timestamp sequence as the global master clock reference of the system. The audio feature extraction submodule is used to convert the classroom video file into a time-stamped text sequence and an interactive tone parameter sequence.

3. The classroom interaction annotation system based on time-point jump according to claim 2, characterized in that, The media stream demultiplexing unit in the audio and video data synchronization acquisition and alignment module supports the standard digital container format, and the main time base extraction unit extracts the timestamp sequence from the container's metadata or data packet header information.

4. The classroom interaction annotation system based on time-point jump according to claim 2, characterized in that, The audio feature extraction submodule uses a deep learning-based speech recognition and tone analysis algorithm model to convert the classroom video file into a timestamped text sequence and an interactive tone parameter sequence.

5. The classroom interaction annotation system based on time-point jump according to claim 1, characterized in that, The rule base of the tag generation module adopts a hybrid mechanism based on semantic templates and keyword matching. The semantic templates define the contextual patterns of interaction types, and the keyword matching library contains the trigger word set corresponding to each interaction type.

6. The classroom interaction annotation system based on time-point jump according to claim 1, characterized in that, The cross-modal feature fusion and label representation module adopts an attention-based multimodal fusion network, which includes a text encoder, an audio encoder, and a video encoder. The text encoder uses a bidirectional long short-term memory network to process text sequences, the audio encoder uses a one-dimensional convolutional network to process tone parameter sequences, and the video encoder uses a three-dimensional convolutional network to process video frame sequences.

7. The classroom interaction annotation system based on time-point jump according to claim 1, characterized in that, The multi-dimensional tag quantization and jump module includes a tag sequence organization unit and a jump instruction generation unit. The tag sequence organization unit is used to sort the tags in chronological order, and the jump instruction generation unit converts the sorted tag sequence into a list of clickable elements.

8. The classroom interaction annotation system based on time-point jump according to claim 1, characterized in that, The adaptive visualization and interactive navigation interface module integrates a video player, a tag list area, and a timeline visualization area. The video player supports H.264 and VP9 encoding formats. The tag list area displays a list of clickable tags, with the list items formatted as hour, minute, second, and interaction type. The timeline visualization area overlays a tag distribution map onto the video player's timeline, with different colors used to mark different interaction types.

9. A classroom interaction annotation method based on time-point jumps, characterized in that, The method includes the following steps: The system receives classroom recording files and parses and decodes them through the audio and video data synchronization acquisition and alignment module. It extracts basic audio and video feature data streams and text sequences that are precisely aligned with the global master clock reference. The parsing process includes media stream demultiplexing, timestamp extraction, and audio feature conversion. The text sequence is transformed into a structured interactive label sequence aligned with the global master clock reference according to preset rules; The basic feature data streams of audio and video are fused to generate a unified fused label vector sequence; The interactive label sequence is classified and quantized, and the final label sequence and jump signal sequence are output. The adaptive visualization and interactive navigation interface module transforms the tag sequence into a clickable list and responds to user clicks to achieve precise navigation to audio and video playback points.

10. An electronic device, characterized in that, include: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes these computer instructions to perform the classroom interactive annotation method based on time jumps as described in claim 9.

Images