Classroom video segmentation and directory creation method based on multi-modal information

Abstract

The invention relates to the technical field of intelligent education video processing, and discloses a classroom video segmentation and directory creation method based on multi-modal information. According to the method, video pictures, audio and screen text signals are collected synchronously, picture change, voice energy contour and text keyword flash features are extracted in a framing mode after time domain alignment, the features are fused into mixed features, and classroom rhythm sudden change points are detected according to the evolution track of the mixed features so as to conduct initial segmentation. And calculating the content fingerprint of each video paragraph, constructing a paragraph association network diagram, identifying paragraph groups with closely associated semantics, recombining the paragraph groups into coherent knowledge units, and finally generating a structured video directory according to the time stamps of the knowledge units. According to the scheme, a teaching content conversion boundary is accurately positioned through multi-modal feature collaborative analysis, and cross-period associated content is recombined by using a semantic network, so that automatic video directory construction conforming to cognitive logic is realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent educational video processing technology, specifically a method for classroom video segmentation and catalog creation based on multimodal information. Background Technology

[0002] Currently, structured processing of classroom videos mainly relies on the analysis of single-modal information. Common approaches include visual scene transition detection, identification of silent segments or energy mutation points based on audio, and simple keyword matching and timestamp annotation of screen text. These techniques typically operate independently, performing preliminary segmentation of the video stream based solely on threshold changes in their own signal dimensions.

[0003] Such schemes based on single-modal threshold discrimination have shortcomings. The shifts in classroom teaching rhythm are the result of multiple elements, including the teacher's language, blackboard writing, presentation slides, and student attention. A single, drastic change in visuals or sound is insufficient to accurately represent the start and end of a knowledge point. Short pauses in audio may indicate a pause for thought rather than a chapter transition, and flipping through slides during continuous explanations does not equate to the end of a knowledge unit. Furthermore, simply segmenting video into chronological order often disrupts the discontinuous temporal distribution of content on the same topic. Teachers may review, deepen, or supplement a core concept multiple times during a lesson, resulting in related teaching content being scattered across different time periods. Existing linear segmentation methods cannot effectively aggregate and reorganize these semantically related but temporally dispersed segments.

[0004] There is a need for a technology that can comprehensively utilize multi-source information in the classroom, accurately perceive substantial turning points in the teaching rhythm, and intelligently identify and reorganize teaching content that is discrete in time but highly related in semantics, thereby forming a knowledge unit system that conforms to cognitive logic and providing a foundation for generating high-quality video catalogs. Summary of the Invention

[0005] The purpose of this invention is to provide a method for classroom video segmentation and catalog creation based on multimodal information, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, this invention provides a method for classroom video segmentation and catalog creation based on multimodal information, the method comprising: Visual image signals, audio signals, and screen text signals from classroom videos are acquired in parallel. The visual image signals, audio signals, and screen text signals are then time-domain aligned to form a synchronous multimodal signal stream. The synchronous multimodal signal stream is divided into frames. For each time frame, the image change features, speech energy contour features, and text keyword flashing features are extracted. The image change features, speech energy contour features, and text keyword flashing features are then merged into the hybrid features of the current frame. Based on the evolution trajectory of the mixed features of each time frame, the abrupt boundary points of the classroom rhythm are detected, and the classroom video is divided into several original video segments based on the abrupt boundary points. Calculate a segment fingerprint for each original video segment, construct a segment association network graph based on the cosine distance between the segment fingerprints of all original video segments, and identify dense subgraphs in the segment association network graph; The dense subgraph is mapped back to the time domain, and the corresponding original video segments are reorganized and spliced ​​to generate semantically coherent knowledge units. Using the start and end times of the knowledge units, segmentation points are marked on the original classroom videos, and a hierarchical video catalog draft is generated based on the segmentation points.

[0007] Preferably, the step of detecting abrupt boundary points of classroom rhythm based on the evolution trajectory of the mixed features of each time frame includes: Slide an analysis window on the time axis to calculate the principal components of the mixture features of all time frames within the current analysis window; Track the rotation angle of the principal component between consecutive analysis windows. When the rotation angle exceeds a preset angle change threshold, record the center point of the current analysis window as a singularity to be verified. Extend a verification interval before and after the singularity to be verified, and calculate the mixed feature mean vector of the two parts before and after the verification interval respectively. Calculate the Mahalanobis distance between the mixed feature mean vectors. When the Mahalanobis distance is greater than a preset distance confirmation threshold, determine that the singular point to be verified is a valid boundary point of classroom rhythm change.

[0008] Preferably, the step of dividing the classroom video into several original video segments based on the mutation boundary points includes: Arrange all valid classroom rhythm change boundary points in chronological order, and extract the video stream between two adjacent change boundary points to form an original video segment; Check the duration of each original video segment. If its duration is less than the preset shortest segment threshold, then forcibly merge the original video segment with its subsequent original video segments. If the duration of the merged segment exceeds the preset maximum segment threshold, then within the merged segment, a secondary division is performed based on the lowest point of the speech energy profile feature to ensure that the duration of each final original video segment is within the preset range.

[0009] Preferably, calculating a segment fingerprint for each original video segment includes: The segment fingerprint is constructed based on a statistical histogram of mixed features within the corresponding original video segment; The image change features of all time frames in an original video segment are concatenated into a feature matrix. Wavelet transform is performed on the feature matrix in the time dimension, and low-frequency wavelet coefficients are extracted as steady-state features of the image. The proportion of speech energy contour features exceeding the activity threshold within the original video segment is counted and used as the speech activity feature. Collect all the text keywords that flash within the original video segment to form a keyword set, and transform the keyword set into a weighted average vector based on pre-trained word vectors as text semantic features; The steady-state features of the video, the activity features of the speech, and the semantic features of the text are standardized and concatenated to form a segment fingerprint representing the original video segment.

[0010] Preferably, the step of constructing a segment association network graph based on the cosine distance between the segment fingerprints of all original video segments includes: Using each original video segment as a network node, calculate the cosine similarity between the segment fingerprints corresponding to any two network nodes. Set a similarity link threshold. When the cosine similarity between two network nodes is higher than the similarity link threshold, establish an undirected edge between the two network nodes. The weight value of the edge is the cosine similarity. By traversing all network node pairs, undirected edges are established, thereby forming a weighted segment association network graph, which reflects the proximity relationships of the original video segments in the feature space.

[0011] Preferably, identifying dense subgraphs in the paragraph association network graph includes: A graph clustering algorithm is run on the weighted paragraph association network graph to aggregate tightly connected network nodes with high edge weights into a community. Calculate the ratio of the average internal edge weight to the average edge weight between communities for each community, and mark communities whose ratio is greater than the community cohesion threshold as candidate dense subgraphs; For each candidate dense subgraph, check the continuity of the original video segments corresponding to the network nodes it contains on the time axis; if the network nodes are not continuous, the candidate dense subgraph is split into multiple time-continuous segments in chronological order, and each time-continuous segment is a final dense subgraph.

[0012] Preferably, the step of mapping the dense subgraph back to the time domain and reorganizing and splicing the corresponding original video segments to generate semantically coherent knowledge units includes: Arrange the network nodes contained in each dense subgraph in order on the timeline according to the timestamp of the original video segment they represent; The original video segments are joined together end to end to form a long video segment sequence. At the start and end positions of the long video segment sequence, the stationarity of the mixed features is checked. If the features at the start position are not stationary, the start point is slightly adjusted forward to a stationary position. If the features at the end position are not stationary, the end point is slightly adjusted backward to a stationary position, thereby determining the precise start and end times of the knowledge unit. Each long video segment sequence with precise start and end times is defined as a knowledge unit.

[0013] Preferably, generating a hierarchical video directory draft based on the segmentation points includes: Each knowledge unit is treated as a leaf node, and its title is generated by extracting frequently occurring screen text keywords within the knowledge unit. Analyze the temporal adjacency and paragraph fingerprint similarity relationships between knowledge units, group multiple knowledge units that are temporally consecutive and whose paragraph fingerprint similarity exceeds the merging threshold into a parent node, and generate a summary title for the parent node; The merge operation is performed recursively until no new parent node can be formed, thereby building a multi-level directory tree from the bottom up, i.e., a hierarchical video directory draft.

[0014] Preferably, the step of sliding an analysis window on the time axis and calculating the principal components of the mixture features of all time frames within the current analysis window includes: Set an analysis window with a fixed time length, and slide the analysis window on the time axis with a fixed step size.

[0015] For each analysis window position, extract the mixed features of all time frames falling within the analysis window, and arrange the mixed features in chronological order to form a feature matrix.

[0016] The feature matrix is ​​centered by subtracting the mean of each feature dimension to obtain the centered feature matrix.

[0017] Calculate the covariance matrix of the centered feature matrix, and then perform eigenvalue decomposition on the covariance matrix.

[0018] Select the top few eigenvectors with the largest eigenvalues ​​and use them as principal components to represent the main direction of the evolution of mixed features within the current analysis window.

[0019] Preferably, the step of running a graph clustering algorithm on the weighted paragraph association network graph to aggregate tightly connected network nodes with high edge weights into a community includes: Initialize the paragraph association network graph, treating each network node as an independent community.

[0020] Calculate the modularity gain resulting from merging all adjacent communities; this modularity gain is used to measure the quality of community division.

[0021] Find and execute the community merge operation that brings the greatest modularity gain, and merge the corresponding communities.

[0022] The process of repeatedly calculating the modularity gain and performing the merging operation continues until the modularity of the entire network no longer increases.

[0023] The resulting community partitioning is the final community, with each community containing a group of closely connected network nodes with high edge weights.

[0024] Compared with the prior art, the beneficial effects of the present invention are: By acquiring and temporally aligning visual, speech, and text signals in parallel, the system extracts visual changes, speech energy contours, and text keyword flashing features for each time frame, merging these three into a hybrid feature. Based on the evolution trajectory of this hybrid feature, it detects abrupt boundary points of classroom rhythm changes, achieving multi-dimensional collaborative perception of teaching transitions. This approach overcomes the shortcomings of single-modal signals, such as high noise and weak indicativeness, and can more accurately capture the real teaching rhythm changes constituted by language emphasis, content switching, and visual cues, making the initial segmentation points of the video more consistent with actual knowledge point boundaries.

[0025] For each initially segmented original video segment, a segment fingerprint representing its content features is calculated, and a segment association network graph is constructed based on this fingerprint. Groups of semantically closely related segments are discovered by identifying dense subgraphs within the graph. These dense subgraphs are mapped back to the time domain, and the corresponding original segments are reorganized and spliced ​​together to generate semantically coherent knowledge units. This method overcomes the limitations of traditional linear processing based on chronological order, automatically aggregating content on the same topic taught by teachers at different times. This ensures the semantic integrity of the final generated knowledge units, allowing the created catalog to reflect the inherent logical structure of the course content rather than a simple chronological listing. Attached Figure Description

[0026] Figure 1 This is a schematic diagram illustrating the working principle of the classroom video segmentation and catalog creation method based on multimodal information described in this invention. Figure 2A flowchart for detecting abrupt boundary points in classroom rhythm; Figure 3 A flowchart for dividing a classroom video into several original video segments; Figure 4 The modularity gain and the dynamic changes in the number of communities during the community merging process; Figure 5 This is a statistical chart showing the duration distribution of classroom video hierarchical directory nodes. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Please see Figure 1 This invention provides a method for classroom video segmentation and catalog creation based on multimodal information. The method includes: parallel acquisition of visual image signals, audio signals, and screen text signals from a classroom video; temporal alignment of the visual image signals, audio signals, and screen text signals using timestamps to form a synchronous multimodal signal stream; frame-by-frame processing of the synchronous multimodal signal stream; extraction of image change features, audio energy contour features, and text keyword flashing features for each time frame; merging the image change features, audio energy contour features, and text keyword flashing features into a mixed feature for the current frame; detecting abrupt boundary points of classroom rhythm based on the evolution trajectory of the mixed features of each time frame; segmenting the classroom video into several original video segments based on these boundary points; calculating a segment fingerprint for each original video segment; constructing a segment association network graph based on the cosine distance between the segment fingerprints of all original video segments; and identifying dense subgraphs in the segment association network graph; mapping the dense subgraphs back to the time domain; and recombining and splicing the corresponding original video segments to generate semantically coherent knowledge units. By using the start and end times of knowledge units, dividing points are marked on the original classroom videos, and a hierarchical video catalog draft is generated based on these dividing points.

[0029] In one embodiment of the present invention, see [reference] Figure 2The process involves sliding an analysis window across the time axis and calculating the principal components of the mixed features across all time frames within that window. A fixed-length analysis window is defined and slid across the time axis with a fixed step size. For each analysis window position, the mixed features of all time frames falling within that window are extracted. These mixed features are arranged chronologically to form a feature matrix. The feature matrix is ​​then centered by subtracting the mean of each feature dimension. The covariance matrix of the centered feature matrix is ​​calculated, and eigenvalue decomposition is performed. The top few eigenvectors with the largest eigenvalues ​​are selected as principal components to represent the main direction of the mixed feature evolution within the current analysis window. The rotation angle of the principal components between consecutive analysis windows is tracked. When the rotation angle exceeds a preset abrupt change threshold, the center point of the current analysis window is recorded as a singularity to be verified. Extend a verification interval before and after the singularity to be verified, calculate the mean vector of the mixed features of the two parts before and after the verification interval, calculate the Mahalanobis distance between the mean vectors of the mixed features, and determine the singularity to be verified as a valid boundary point of classroom rhythm change when the Mahalanobis distance is greater than the preset distance confirmation threshold.

[0030] In practice, the multimodal signal stream of the classroom video has undergone temporal alignment and framing. Each time frame has had its image change features, speech energy contour features, and text keyword flashing features extracted and merged to form a hybrid feature vector. Taking a 45-minute math class video as an example, the hybrid feature vector has a dimension of 128. The detection process begins by sliding an analysis window along the timeline. The fixed duration of the analysis window is set to 90 seconds, and the sliding step is set to 10 seconds. For each analysis window position, the hybrid features of all time frames falling within the window are extracted. For example, an analysis window located between seconds 300 and 390 of the video contains 540 time frames. The hybrid features of these time frames are arranged chronologically to form a 540-row, 128-column feature matrix. This feature matrix is ​​then centered by calculating the mean of the 128-dimensional features in each column and subtracting this mean from all values ​​in that column to obtain the centered feature matrix.

[0031] The covariance matrix of the centered feature matrix is ​​calculated. This covariance matrix is ​​a 128x128 square matrix, and then eigenvalue decomposition is performed. The top three eigenvectors with the largest eigenvalues ​​are selected as principal components. These three principal component vectors collectively represent the main direction of the evolution of the mixed features within the current analysis window. The sliding analysis window traverses the entire video timeline, repeating the above principal component calculation process for each window position, thus obtaining a set of principal component vectors that change over time. The rotation angle of the principal components between consecutive analysis windows is tracked. The rotation angle is obtained by calculating the angle between the principal component vectors corresponding to adjacent windows. An angle abrupt change threshold of 45 degrees is set. When the calculated rotation angle exceeds 45 degrees, the center point of the current analysis window is recorded as the singularity to be verified. A verification interval is extended before and after the singularity to be verified. The time length of the verification interval is set to 60 seconds. Taking the singularity at the aforementioned 455-second mark as an example, the verification interval is from the 425-second mark to the 485-second mark. Calculate the mean vector of the mixed features in the two parts of the verification interval separately. That is, average the mixed features of all time frames in the interval from 425 seconds to 455 seconds to obtain the mean vector of the first half, and average the mixed features of all time frames in the interval from 455 seconds to 485 seconds to obtain the mean vector of the second half.

[0032] In one embodiment of the present invention, see [reference] Figure 3 All valid classroom rhythm abrupt change boundary points are arranged chronologically. Video streams are extracted between two adjacent boundary points to form an original video segment. The duration of each original video segment is checked. If its duration is less than a preset minimum segment threshold, the original video segment is forcibly merged with its subsequent original video segments. If the duration of the merged segment exceeds a preset maximum segment threshold, the merged segment is further divided based on the lowest point of the speech energy profile features to ensure that the duration of each final original video segment is within a preset range. A segment fingerprint is calculated for each original video segment, based on a statistical histogram of the mixed features within the corresponding original video segment. The image change features of all time frames within an original video segment are concatenated into a feature matrix. Wavelet transform is applied to the feature matrix in the time dimension to extract low-frequency wavelet coefficients as image steady-state features. The proportion of speech energy contour features exceeding the activity threshold within the original video segment is counted as speech activity features. All flashing text keywords within the original video segment are collected to form a keyword set. The keyword set is transformed into a weighted average vector based on pre-trained word vectors as text semantic features. The image steady-state features, speech activity features, and text semantic features are standardized and concatenated to form a segment fingerprint representing the original video segment.

[0033] In practice, assuming that the effective classroom rhythm change boundary points detected in a 2700-second classroom video are arranged in chronological order as follows: 455 seconds, 1020 seconds, 1780 seconds, and 2310 seconds. After arranging all effective classroom rhythm change boundary points in chronological order, the video stream is segmented between two adjacent change boundary points to form an original video segment. Specifically, the first original video segment is segmented from the video start point 0 seconds to the first boundary point 455 seconds; the second original video segment is segmented from 455 seconds to 1020 seconds; the third original video segment is segmented from 1020 seconds to 1780 seconds; the fourth original video segment is segmented from 1780 seconds to 2310 seconds; and the fifth original video segment is segmented from 2310 seconds to the end of the video at 2700 seconds.

[0034] The duration of each original video segment is checked, with a preset minimum segment threshold of 120 seconds and a preset maximum segment threshold of 600 seconds. The first original video segment lasts 455 seconds, and the second original video segment lasts 565 seconds; both of these durations are within the preset range. The fourth original video segment lasts 530 seconds, and the fifth original video segment lasts 390 seconds; both of these durations are also within the preset range. The third original video segment lasts 760 seconds, which exceeds the preset maximum segment threshold of 600 seconds. Therefore, it is necessary to perform a secondary segmentation within the third original video segment based on the lowest point of the speech energy profile features. The speech energy profile from 1020 seconds to 1780 seconds within the third original video segment is analyzed to find the lowest point of the speech energy profile features within this interval, for example, at 1420 seconds. At the lowest point of 1420 seconds, the third original video segment was divided into two sub-segments: one from 1020 seconds to 1420 seconds, and the other from 1420 seconds to 1780 seconds. The durations of the two sub-segments after the division were 400 seconds and 360 seconds, respectively, both within the preset duration range.

[0035] The proportion of speech energy contour features exceeding an activity threshold within the original video segments is used as the speech activity feature. The activity threshold is set to 1.5 times the mean of the speech energy contour features within that segment. In the first segment, the proportion of frames exceeding the activity threshold is 0.68, therefore the speech activity feature is a scalar value of 0.68. All fleeting text keywords within the original video segments are collected to form a keyword set; for example, the keyword set for the first segment includes "function," "domain," and "mapping." The keyword set is transformed into a weighted average vector based on pre-trained word vectors, serving as the text semantic feature. The pre-trained word vector model is GloVe, with a word vector dimension of 300. The weight of each keyword is determined by its frequency of occurrence within the segment, resulting in a 300-dimensional text semantic feature.

[0036] The steady-state features of the video, the speech activity features, and the text semantic features are standardized and concatenated to form a segment fingerprint representing the original video segment. Standardization uses the Z-score method, standardizing the 128-dimensional steady-state feature vector, the 1-dimensional speech activity feature scalar, and the 300-dimensional text semantic feature vector respectively. The concatenated segment fingerprint is a 429-dimensional feature vector, calculated using the following formula: Where: symbol Represents the segment fingerprint vector, symbol Represents Z-score normalization, symbol Represents the steady-state feature vector of the image, symbol Represents a scalar value for speech activity features, with the symbol... Represents the semantic feature vector of the text, symbols This represents the cascading operation of vectors.

[0037] In one embodiment of the present invention, each original video segment is used as a network node, and the cosine similarity between the segment fingerprints corresponding to any two network nodes is calculated. A similarity linking threshold is set. When the cosine similarity between two network nodes is higher than the similarity linking threshold, an undirected edge is established between the two network nodes, and the weight value of the edge is the cosine similarity. All network node pairs are traversed to complete the establishment of undirected edges, thereby forming a weighted segment association network graph, which reflects the proximity relationship of the original video segments in the feature space.

[0038] In practice, the classroom video is segmented and merged into six original video segments, denoted as Segment A, Segment B, Segment C, Segment D, Segment E, and Segment F. Each segment's fingerprint is a 429-dimensional feature vector. A segment association network graph is constructed based on the cosine distance between the segment fingerprints of all the original video segments. This process is illustrated using these six segments as an example. Constructing the segment association network graph begins with each original video segment as a network node; the six original video segments correspond to six network nodes. The cosine similarity between the segment fingerprints of any two network nodes is calculated using the standard inner product formula. For network node i and network node j, their corresponding segment fingerprint vectors are respectively... and The similarity calculation formula is: Where: symbol The cosine similarity value between network node i and network node j is represented by the symbol. The segment fingerprint vector corresponding to network node i, symbol The segment fingerprint vector corresponding to network node j, with the symbol... The dot product operation represents vectors, with the sign... It represents the L2 norm of a vector.

[0039] In practice, the cosine similarity between all pairs of network nodes is calculated, forming a 6x6 symmetric similarity matrix. The values ​​on the main diagonal of the matrix represent the similarity between a network node and itself, with a value of 1. A similarity link threshold is set to 0.85. When the cosine similarity between two network nodes is higher than the similarity link threshold of 0.85, an undirected edge is established between the two network nodes. Taking network node A and network node B as an example, the segment fingerprint vector is calculated. and cosine similarity Since 0.92 is greater than 0.85, an undirected edge is established between network node A and network node B. The weight of this undirected edge is the cosine similarity value of 0.92.

[0040] In some embodiments, the similarity link threshold can be adjusted based on the distribution of all cosine similarity values. The similarity link threshold can be set to the median or upper quartile of all similarity values. All network node pairs are traversed to establish undirected edges. For example, if the similarity between network node C and network node D is 0.78, which is below the threshold, no connection is established; if the similarity between network node D and network node E is 0.88, which is above the threshold, an edge with a weight of 0.88 is established. After traversal, a weighted segment association network graph is finally formed. This segment association network graph is an undirected weighted graph, where nodes represent original video segments, and edges represent high similarity relationships of segment fingerprints in the feature space. It can be understood that the segment association network graph intuitively reflects the proximity relationships of original video segments in the feature space in the form of a graph structure, with high-weighted edges connecting segments with similar feature expressions.

[0041] In one embodiment of the invention, a graph clustering algorithm is run on a weighted segment association network graph to aggregate tightly connected network nodes with high edge weights into a community. The segment association network graph is initialized, treating each network node as an independent community. The modularity gain resulting from merging all adjacent communities is calculated. This modularity gain measures the quality of the community partitioning. The community merging operation that yields the maximum modularity gain is found and executed. The corresponding communities are merged, and the process of calculating the modularity gain and executing the merging operation is repeated until the modularity of the entire network no longer increases. The resulting community partitioning is the final community, where each community contains a group of tightly connected network nodes with high edge weights. The ratio of the average internal edge weight to the average inter-community edge weight is calculated for each community. Communities with a ratio greater than a community cohesion threshold are marked as candidate dense subgraphs. For each candidate dense subgraph, the continuity of the original video segments corresponding to its network nodes on the timeline is checked. If the network nodes are not continuous, the candidate dense subgraph is split into multiple temporally continuous segments, each of which is a final dense subgraph.

[0042] In practical implementation, network nodes correspond to six original video segments A, B, C, D, E, and F. The edges between nodes and their weights represent the similarity of segment fingerprints. Identifying dense subgraphs in the segment association network graph requires running a graph clustering algorithm on the weighted segment association network graph to aggregate tightly connected network nodes with high edge weights into a community. The segment association network graph is initialized, treating each network node as an independent community; therefore, the initial state has six communities: {A}, {B}, {C}, {D}, {E}, and {F}. The modularity gain generated after merging all adjacent communities is calculated; this modularity gain measures the quality of community partitioning. The formula for calculating the modularity increment ΔQ is: Where: symbol The moduli gain resulting from merging community i and community j is represented by the symbol. The symbol represents the sum of the weights of all edges between community i and community j. The symbol represents the sum of the weights of all edges connecting nodes in community i. The symbol represents the sum of the weights of all edges connecting nodes in community j. It represents half of the total weight of all edges in the entire paragraph's network graph.

[0043] Find and execute the community merge operation that yields the maximum modularity gain, merging the corresponding communities. Assume that among all possible community merges, merging community {A} with community {B} yields the greatest modularity gain. The modularity gain is 0.12, which is the maximum value among all candidate merges. Therefore, the merge operation is performed, forming a new community {A,B}. The process of calculating the modularity gain and performing merge operations is repeated until the modularity of the entire network no longer increases. In subsequent merges, communities {D} and {E} may be merged, and then communities {C} and {A,B} may be merged. The resulting community partitioning is the final community, with each community containing a group of tightly connected network nodes with high edge weights. Assume that three communities are ultimately obtained: Community 1 contains nodes {A,B,C}, Community 2 contains nodes {D,E}, and Community 3 contains nodes {F}.

[0044] Calculate the ratio of the average internal edge weight to the average inter-community edge weight for each community. Communities with a ratio greater than the community cohesion threshold are marked as candidate dense subgraphs. The community cohesion threshold is set to 2.0. It is necessary to calculate the average internal edge weight for each community, as well as the average edge weight between that community and all other communities. Community 1 contains nodes A, B, and C; its internal edge connections are shown in Table 1. Table 1: Internal Edge Connections of Community Nodes According to the data in the table above, the average internal edge weight of Community 1 is (0.92 + 0.90 + 0.88) / 3 = 0.90. The total weight of the edges between Community 1 and other communities is 0.70 + 0.60 + 0.65 = 1.95. Since there are 3 edges between communities, the average weight of the edges between communities is 1.95 / 3 = 0.65. The ratio of the average internal edge weight of Community 1 to the average weight of the edges between communities is 0.90 / 0.65 ≈ 1.38. Because 1.38 is less than the community cohesion threshold of 2.0, Community 1 is not marked as a candidate dense subgraph. The average internal edge weight of Community 2 is 0.89. The total weight of the edges between communities in Community 2 is 0.70 + 0.65 = 1.35. Since there are 2 edges between communities, the average weight of the edges between communities is 1.35 / 2 = 0.675. The ratio for Community 2 is 0.89 / 0.675 ≈ 1.32, which is also less than the threshold of 2.0. Community 3 contains only one node F and has no internal edges; its average internal edge weight cannot be calculated, therefore it does not meet the criteria to be a candidate dense subgraph. It can be understood that if the average strength of internal connections within a community is significantly higher than the average strength of its external connections, it indicates that the nodes within that community form a relatively tight group. In some embodiments, the community cohesion threshold can be adjusted based on the average clustering coefficient of the specific network.

[0045] For each candidate dense subgraph, examine the temporal continuity of the original video segments corresponding to its contained network nodes. Assume there exists a candidate dense subgraph containing network nodes {B, D, E}, whose corresponding original video segments have timestamps of segments B (455s-1020s), D (1420s-1780s), and E (1780s-2310s), respectively. The original video segments corresponding to nodes B, D, and E are not contiguous on the timeline; there is a time gap between segments B and D, while segments D and E are temporally continuous. The candidate dense subgraph is then split into multiple temporally continuous segments, each of which constitutes a final dense subgraph. Therefore, the candidate dense subgraph {B, D, E} is split into two final dense subgraphs: one containing only node B, and the other containing the temporally continuous nodes {D, E}.

[0046] See Figure 4During the community merging process, the dynamic changes in modularity gain and the number of communities in the paragraph association network graph show the evolution of modularity gain and the number of communities in stage 2 as the merging steps progress. Specifically, the initial state corresponds to 6 independent communities ({A}, {B}, {C}, {D}, {E}, {F}), with a modularity gain of 0. After the first merge (merging {A} and {B}), the modularity gain reaches a peak of 0.12, and the number of communities decreases to 5. In subsequent merge steps (such as merging {D} and {E}), the modularity gain drops to 0.08, and the number of communities further decreases to 4. As the merging operation continues (such as merging {C} and {A,B}), the modularity gain gradually decreases, and the number of communities is adjusted to 3. In the final state, the modularity gain falls back to 0, and the number of communities stabilizes at 3 (corresponding to communities {A,B,C}, {D,E}, and {F}). It intuitively reflects the dynamic pattern that "the modularity gain decreases with the number of mergers and the number of communities gradually converges with the number of merger operations" during the community merging process, providing a visual basis for evaluating the quality of community division.

[0047] In one embodiment of the present invention, the network nodes contained in each dense subgraph are arranged sequentially on the timeline according to the timestamps of the original video segments they represent. The arranged original video segments are then joined end-to-end to merge them into a long video segment sequence. At the start and end positions of the long video segment sequence, the mixed features are tested for stationarity. If the features at the start position are not stationary, the start point is finely adjusted forward to a stationary position; if the features at the end position are not stationary, the end point is finely adjusted backward to a stationary position, thereby determining the precise start and end times of the knowledge unit. Each long video segment sequence with precise start and end times is defined as a knowledge unit. A hierarchical video catalog draft is generated based on the segmentation points, including the following process: each knowledge unit is treated as a leaf node, and its title is generated by extracting frequently occurring screen text keywords within the knowledge unit. The temporal adjacency relationship and segment fingerprint similarity relationship between knowledge units are analyzed. Multiple knowledge units that are temporally continuous and whose segment fingerprint similarity exceeds the merging threshold are merged into a parent node, and a summary title is generated for the parent node. The merge operation is performed recursively until no new parent node can be formed, thereby building a multi-level directory tree from the bottom up, which is a hierarchical video directory draft.

[0048] In practice, the dense subgraph is mapped back to the time domain, and the corresponding original video segments are reorganized and spliced ​​to generate semantically coherent knowledge units. This process begins by arranging the network nodes contained in each dense subgraph sequentially on the timeline according to the timestamps of the original video segments they represent. For a dense subgraph containing network nodes {D, E}, the timestamps of the original video segments represented by network node D are from 1420 seconds to 1780 seconds, and the timestamps of the original video segments represented by network node E are from 1780 seconds to 2310 seconds. The result of chronological arrangement is that network node D comes first, and network node E comes last. The arranged original video segments are then joined end to end to merge them into a long video segment sequence, that is, starting from 1420 seconds, original video segments D and E are spliced ​​together in sequence to form a long video segment sequence that lasts from 1420 seconds to 2310 seconds.

[0049] At the start and end positions of the long video segment sequence, a stationarity test is performed on the mixed features. The stationarity test is evaluated by calculating the variance of the mixed feature vector within a short time window near the start point of the sequence; for example, examining the mixed features of all frames within a 30-second window from 1420 seconds to 1450 seconds. If the features at the start position are not stationary, the start point is fine-tuned forward until the features are stationary. Assume the variance of the mixed feature vector within the window from 1420 seconds to 1450 seconds is... The preset stationarity variance threshold is ,when At that time, the initial position characteristic was determined to be unstable. Calculation revealed... and Therefore, the starting point is fine-tuned forward. The search proceeds in 1-second increments, checking the variance of the window from 1419 to 1449 seconds. If the variance is 7.8, below the threshold, the starting point of the knowledge unit is fine-tuned to 1419 seconds. The same logic is applied to the ending position; if the feature at the ending position is not stationary, the ending point is fine-tuned backward until the feature is stationary. Assume the variance of the window 30 seconds backward from the initial ending point of 2310 seconds (from 2310 to 2340 seconds) is... If the endpoint is not found, the endpoint is finely adjusted backward, and the variance of the window from second 2311 to second 2341 is checked until the variance is found to be below the threshold. The precise start and end times of the knowledge unit are determined through stationarity testing and fine-tuning. Each long video segment sequence with precise start and end times is defined as a knowledge unit; for example, the start and end times of knowledge unit K1 are ultimately determined to be from second 1419 to second 2315.

[0050] A hierarchical video directory draft is generated based on the segmentation points, including the following processes: Each knowledge unit is treated as a leaf node, and its title is generated by extracting frequently occurring screen text keywords within the knowledge unit. For knowledge unit K1, the frequency of all screen text keywords from 1419 seconds to 2315 seconds is analyzed. The top three most frequent keywords are "derivative," "operational rules," and "composite function." These high-frequency words are combined to generate the leaf node title "derivative operation rules and composite functions." The temporal adjacency and paragraph fingerprint similarity relationships between knowledge units are analyzed. Multiple knowledge units that are temporally continuous and whose paragraph fingerprint similarity exceeds the merging threshold are merged into a single parent node. Assume there is another knowledge unit K2, whose time range is from 950 seconds to 1418 seconds, which is temporally continuous with knowledge unit K1. The paragraph fingerprint similarity between knowledge unit K1 and knowledge unit K2 is calculated, and the merging threshold is set to 0.80. The segment fingerprint of knowledge unit K1 is generated by fusing the segment fingerprints of the original video segments D and E contained therein, and the segment fingerprint of knowledge unit K2 is generated by fusing the fingerprints of the original video segments contained therein. The cosine similarity between the two is calculated. If the result is 0.85, which exceeds the merging threshold of 0.80, then knowledge unit K1 and knowledge unit K2 are merged into a parent node.

[0051] See Figure 5 In the construction and duration analysis of the hierarchical directory of classroom videos, the distribution characteristics of the duration (in seconds) of different level nodes (level 1, level 2, and level 3) under each directory node name are presented intuitively. Specifically, the duration of the level 1 nodes corresponding to "Basic Calculus" and "Advanced Calculus" reaches 3500 seconds and 1800 seconds respectively, reflecting the content carrying capacity of the level 1 nodes as core knowledge modules; the duration of the level 2 nodes "Derivatives and Limits" and "Integral Calculation" is 2300 seconds and 1600 seconds respectively, reflecting the focus of the level 2 nodes on subdivided knowledge topics; the level 3 nodes are distributed in specific knowledge points such as "Derivative Operation Rules" and "Function Limits", with the duration concentrated in the range of 400-900 seconds, which is consistent with the refined characteristics of the level 3 nodes as knowledge units. The difference in duration of nodes at different levels corresponds to the hierarchical structure of knowledge units and parent nodes in the segmentation of classroom videos, and also provides a quantitative basis for evaluating the semantic coherence and duration rationality of the directory.

[0052] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0053] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for classroom video segmentation and catalog creation based on multimodal information, characterized in that, Including the following steps: Visual image signals, audio signals, and screen text signals from classroom videos are acquired in parallel. The visual image signals, audio signals, and screen text signals are then time-domain aligned to form a synchronous multimodal signal stream. The synchronous multimodal signal stream is divided into frames. For each time frame, the image change features, speech energy contour features, and text keyword flashing features are extracted. The image change features, speech energy contour features, and text keyword flashing features are then merged into the hybrid features of the current frame. Based on the evolution trajectory of the mixed features of each time frame, the abrupt boundary points of the classroom rhythm are detected, and the classroom video is divided into several original video segments based on the abrupt boundary points. Calculate a segment fingerprint for each original video segment, construct a segment association network graph based on the cosine distance between the segment fingerprints of all original video segments, and identify dense subgraphs in the segment association network graph; The dense subgraph is mapped back to the time domain, and the corresponding original video segments are reorganized and spliced ​​to generate semantically coherent knowledge units. Using the start and end times of the knowledge units, segmentation points are marked on the original classroom videos, and a hierarchical video catalog draft is generated based on the segmentation points.

2. The method for classroom video segmentation and directory creation based on multimodal information as described in claim 1, characterized in that, The method of detecting abrupt boundary points of classroom rhythm based on the evolution trajectory of the mixed features of each time frame includes: Slide an analysis window on the time axis to calculate the principal components of the mixture features of all time frames within the current analysis window; Track the rotation angle of the principal component between consecutive analysis windows. When the rotation angle exceeds a preset angle change threshold, record the center point of the current analysis window as a singularity to be verified. Extend a verification interval before and after the singularity to be verified, and calculate the mixed feature mean vector of the two parts before and after the verification interval respectively. Calculate the Mahalanobis distance between the mixed feature mean vectors. When the Mahalanobis distance is greater than a preset distance confirmation threshold, determine that the singular point to be verified is a valid boundary point of classroom rhythm change.

3. The method for classroom video segmentation and directory creation based on multimodal information as described in claim 1, characterized in that, The classroom video is divided into several original video segments based on the mutation boundary points, including: Arrange all valid classroom rhythm change boundary points in chronological order, and extract the video stream between two adjacent change boundary points to form an original video segment; Check the duration of each original video segment. If its duration is less than the preset shortest segment threshold, then forcibly merge the original video segment with its subsequent original video segments. If the duration of the merged segment exceeds the preset maximum segment threshold, then within the merged segment, a secondary division is performed based on the lowest point of the speech energy profile feature to ensure that the duration of each final original video segment is within the preset range.

4. The method for classroom video segmentation and directory creation based on multimodal information as described in claim 1, characterized in that, The step of calculating a segment fingerprint for each original video segment includes: The segment fingerprint is constructed based on a statistical histogram of mixed features within the corresponding original video segment; The image change features of all time frames in an original video segment are concatenated into a feature matrix. Wavelet transform is performed on the feature matrix in the time dimension, and low-frequency wavelet coefficients are extracted as steady-state features of the image. The proportion of speech energy contour features exceeding the activity threshold within the original video segment is counted and used as the speech activity feature. Collect all the text keywords that flash within the original video segment to form a keyword set, and transform the keyword set into a weighted average vector based on pre-trained word vectors as text semantic features; The steady-state features of the video, the activity features of the speech, and the semantic features of the text are standardized and concatenated to form a segment fingerprint representing the original video segment.

5. The method for classroom video segmentation and directory creation based on multimodal information as described in claim 1, characterized in that, The step of constructing a segment association network graph based on the cosine distance between the segment fingerprints of all original video segments includes: Using each original video segment as a network node, calculate the cosine similarity between the segment fingerprints corresponding to any two network nodes. Set a similarity link threshold. When the cosine similarity between two network nodes is higher than the similarity link threshold, establish an undirected edge between the two network nodes. The weight value of the edge is the cosine similarity. By traversing all network node pairs, undirected edges are established, thereby forming a weighted segment association network graph, which reflects the proximity relationships of the original video segments in the feature space.

6. The method for classroom video segmentation and directory creation based on multimodal information as described in claim 5, characterized in that, The identification of dense subgraphs in the paragraph association network graph includes: A graph clustering algorithm is run on the weighted paragraph association network graph to aggregate tightly connected network nodes with high edge weights into a community. Calculate the ratio of the average internal edge weight to the average edge weight between communities for each community, and mark communities whose ratio is greater than the community cohesion threshold as candidate dense subgraphs; For each candidate dense subgraph, check the continuity of the original video segments corresponding to the network nodes it contains on the time axis; if the network nodes are not continuous, the candidate dense subgraph is split into multiple time-continuous segments in chronological order, and each time-continuous segment is a final dense subgraph.

7. The method for classroom video segmentation and directory creation based on multimodal information as described in claim 6, characterized in that, The step of mapping the dense subgraph back to the time domain, reorganizing and splicing the corresponding original video segments, and generating semantically coherent knowledge units includes: Arrange the network nodes contained in each dense subgraph in order on the timeline according to the timestamp of the original video segment they represent; The original video segments are joined together end to end to form a long video segment sequence. At the start and end positions of the long video segment sequence, the stationarity of the mixed features is checked. If the features at the start position are not stationary, the start point is slightly adjusted forward to a stationary position. If the features at the end position are not stationary, the end point is slightly adjusted backward to a stationary position, thereby determining the precise start and end times of the knowledge unit. Each long video segment sequence with precise start and end times is defined as a knowledge unit.

8. The method for classroom video segmentation and directory creation based on multimodal information as described in claim 1, characterized in that, The step of generating a hierarchical video directory draft based on the segmentation points includes: Each knowledge unit is treated as a leaf node, and its title is generated by extracting frequently occurring screen text keywords within the knowledge unit. Analyze the temporal adjacency and paragraph fingerprint similarity relationships between knowledge units, group multiple knowledge units that are temporally consecutive and whose paragraph fingerprint similarity exceeds the merging threshold into a parent node, and generate a summary title for the parent node; The merge operation is performed recursively until no new parent node can be formed, thereby building a multi-level directory tree from the bottom up, i.e., a hierarchical video directory draft.

9. The method for classroom video segmentation and directory creation based on multimodal information as described in claim 2, characterized in that, The step of sliding an analysis window on the time axis and calculating the principal components of the mixed features of all time frames within the current analysis window includes: Set an analysis window with a fixed time length, and slide the analysis window on the time axis with a fixed step size; For each analysis window position, extract the mixed features of all time frames falling within the analysis window, and arrange the mixed features in chronological order to form a feature matrix; The feature matrix is ​​centered by subtracting the mean of each feature dimension in the feature matrix to obtain the centered feature matrix. Calculate the covariance matrix of the centered feature matrix, and then perform eigenvalue decomposition on the covariance matrix; Select the top few eigenvectors with the largest eigenvalues ​​and use them as principal components to represent the main direction of the evolution of mixed features within the current analysis window.

10. The method for classroom video segmentation and directory creation based on multimodal information as described in claim 6, characterized in that, The graph clustering algorithm, applied to the weighted paragraph association network graph, aggregates tightly connected network nodes with high edge weights into a community, including: Initialize the paragraph association network graph, treating each network node as an independent community; Calculate the modularity gain resulting from merging all adjacent communities; the modularity gain is used to measure the quality of community division. Find and execute the community merge operation that brings the greatest modularity gain, and merge the corresponding communities; Repeat the process of calculating the modularity gain and performing the merging operation until the modularity of the entire network no longer increases; The resulting community partitioning is the final community, with each community containing a group of closely connected network nodes with high edge weights.

Images