Teaching micro-video generation method and system based on semantic and time constraints

Abstract

The invention discloses a teaching micro-video generation method and system based on semantic and time constraints, and relates to the technical field of image processing. Comprising the following steps: performing second-by-second frame extraction on a teaching video, and integrating a plurality of Hash algorithms to perform de-duplication processing so as to obtain a de-duplicated non-repeated slide image of each frame; replacing a Darknet-53 backbone network in the YOLOv8 by using the OfficientViT, introducing an SE attention mechanism, constructing an improved YOLOv8 model, performing identification and slide extraction on each frame of non-repeated slide image after de-duplication by using the improved YOLOv8 model, and obtaining a title of the slide image and a timestamp when the title appears for the first time; the title of the slide image is associated by combining semantics with the title, and an obtained local minimum value is used as a knowledge point boundary; and segmenting the teaching video by using a recursive decision tree algorithm in combination with the knowledge point boundary and a set time window to obtain segmentation results, and merging the segmentation results based on semantic correlation to form a teaching micro-video corresponding to the knowledge points.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for generating teaching micro-videos based on semantic and time constraints. Background Art

[0002] Lecture videos have become the most popular learning resources for students. They are usually organized by chapters and have a duration of about 50 minutes or longer, which makes it inconvenient to quickly locate specific knowledge points. Therefore, the work of generating videos with multi-level tables of contents has emerged. These works mainly focus on video entity recognition and chapter and section table of contents recognition, and the smallest table of contents level is the third-level heading. The method of marking videos with a table of contents may result in relatively large or small abnormal segments, which is still not sufficient for users to quickly locate specific knowledge points. Summary of the Invention

[0003] In order to overcome the above existing drawbacks, the main object of the present invention is to provide a method and system for generating teaching micro-videos based on semantic and time constraints.

[0004] To achieve the above object, the present invention adopts the following technical solutions. A method for generating teaching micro-videos based on semantic and time constraints includes the following steps:

[0005] Extract frames from the teaching video frame by frame per second and perform deduplication processing using multiple hashing algorithms to obtain non-repeating slide images for each frame after deduplication;

[0006] Replace the Darknet-53 backbone network in YOLOv8 with EfficientViT, and introduce the SE attention mechanism to construct an improved YOLOv8 model. Use the improved YOLOv8 model to identify and extract slides from the non-repeating slide images for each frame after deduplication, and obtain the title of the slide image and the timestamp when the title first appears;

[0007] Use a semantic correlation model to associate the title of the slide image with the title, and obtain the local minimum similarity between the titles as the knowledge point boundary;

[0008] Use a recursive decision tree algorithm, combine the knowledge point boundary and the set time window, segment the teaching video to obtain a segmentation result, and merge the segmentation results based on the semantic correlation between the segmentation results to form teaching micro-videos corresponding to the knowledge points.

[0009] The step of using the improved YOLOv8 model to identify and extract slides from each frame of the teaching video includes the following steps:

[0010] Divide the teaching video into a series of consecutive image frames; preprocess each frame of the image to obtain each frame of the processed image;

[0011] The similarity between each processed frame image is obtained by using Average Hash, Difference Hash, Perceptual Hash, and Histogram Similarity algorithms. The redundancy is used to represent the similarity function, and the calculation formula of the redundancy is as follows:

[0012]

[0013] Among them, Θ represents the difference between the hash values of different titles, flag represents the flag bit. When the flag bit is 1, it means the two images are similar; when the flag bit is 0, it means the two images are not similar. A represents the hash value calculated by the Average Hash algorithm; D represents the hash value calculated by the Difference Hash algorithm; P represents the hash value calculated by the Perceptual Hash algorithm; avgHash represents the average value of the three hash values; s1 represents the preset upper threshold, and s2 represents the preset lower threshold. s.t.A represents the similarity measure between the average hash values of the two images, and pic1 and pic2 represent Image 1 and Image 2;

[0014] Associate each frame of slide image of the teaching video with the title by semantic combination, and use the hash algorithm to obtain the local minimum of the similarity between titles as the knowledge point boundary, including the following steps:

[0015] Select the semantic similarity model as ROM;

[0016] Preprocess the title text of each slide to obtain the preprocessed title text;

[0017] Convert the preprocessed title text into a semantic vector representation;

[0018] Associate each slide image with its corresponding title text, and use the timestamp of the slide as the identifier;

[0019] Use the ROM semantic similarity model to obtain the similarity scores between the semantic vectors of adjacent slide titles in sequence, and take the local minimum points on the score curve as the preliminary video segmentation positions;

[0020] Until the local minimum of the similarity between titles is obtained as the knowledge point boundary.

[0021] Using the recursive decision tree algorithm, combining the knowledge point boundary and the set time window, segment the teaching video to obtain the segmentation result, including the following steps:

[0022] Set a time window [θ, η], where θ is the minimum segmentation duration and η is the maximum segmentation duration. Treat the teaching video as a whole as the initial node;

[0023] Select a knowledge point boundary as the segmentation point to divide the initial node into two sub-nodes;

[0024] For each sub-node, when the duration of the set window exceeds the maximum segmentation duration η, divide it;

[0025] When a node contains multiple knowledge points, divide the node containing multiple knowledge points until the duration of the set window of the node is less than the maximum segmentation duration and the node includes one knowledge point to obtain the segmentation result.

[0026] The secondary segmentation of the segmentation result using the equal division strategy includes the following steps:

[0027] According to the segmentation result, obtain the remaining part of the slide image after removing the first-level and second-level headings as the body part;

[0028] Convert the image of the body part into text, count the number of character-level differences in the text of the adjacent slide body parts to obtain the character difference value;

[0029] If the character difference value of the slide exceeds the average value of the adjacent slides, it is determined that the body content has changed substantially. Record the time point and the corresponding body content as the potential knowledge point boundary, and use the recursive decision algorithm combined with the time window constraint to perform secondary segmentation on the nodes not in the time window [θ, η];

[0030] If the character difference value of the adjacent slides is lower than the threshold, it is determined that the content of the video block belongs to the same knowledge point, but its duration exceeds the preset maximum time window. Divide the video block into several sub-blocks so that the duration of each sub-block falls within the allowed time window; if the duration of the sub-block after equal division is less than the minimum time window, merge the sub-block with the adjacent sub-block so that the segmentation result meets the time window constraint to obtain the secondary segmentation result.

[0031] The generation system of the teaching micro-video generation method based on semantic and time constraints includes:

[0032] The teaching video splitting module is used to extract frames from the teaching video frame by frame and perform deduplication processing using multiple hash algorithms to obtain the non-repeated slide images of each frame after deduplication;

[0033] The split image processing module is used to replace the Darknet-53 backbone network in YOLOv8 with EfficientViT, introduce the SE attention mechanism, construct an improved YOLOv8 model, and use the improved YOLOv8 model to identify and extract slides from each non-repeated slide image after duplicate removal, obtaining the title of the slide image and the timestamp when the title first appears; use semantics to associate with the title of the slide image where the title is located, and obtain the local minimum of the similarity between titles as the knowledge point boundary;

[0034] The micro-video synthesis module is used to segment the teaching video using the recursive decision tree algorithm, combine the knowledge point boundary and the set time window to obtain the segmentation result, and merge the segmentation result based on semantic relevance to form teaching micro-videos corresponding to knowledge points.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention first proposes the idea of automatically generating course knowledge point micro-videos from teaching videos and transforms it into a knowledge point-level video segmentation task with semantic and time constraints. This task is further transformed into a slide segmentation task, including three sub-tasks: slide extraction, knowledge point boundary recognition (KBR), and segmentation point recognition based on semantic and time constraints (STC). Among them, the KBR task is transformed into a video entity recognition problem, and the minimum semantic relevance between video entities at the same level is calculated. STC is transformed into a video merging / splitting task with two-stage semantic and time constraints.

[0036] The present invention designs a slide extraction algorithm that integrates average hashing, difference hashing, perceptual hashing, and histogram similarity algorithms. And we propose an enhanced KBR method that combines YOLOv8++ and the MROM model. And use the recursive decision tree algorithm to ensure that the duration of semantic segmentation falls within the time window to achieve appropriate segmentation. YOLOv8++ combines the object detection ability of YOLOv8 with the multi-scale image feature extraction of EfficientViT, improving the object detection accuracy in complex scenarios, especially in video analysis and PPT page extraction. The image duplicate removal technology is adopted to ensure the uniqueness of image content, avoid the influence of duplicate content on subsequent analysis, and improve the efficiency of data processing. Through the object detection and image segmentation technology of YOLOv8, the efficient segmentation of video content is realized, providing a clearer structure for video analysis. The MROM algorithm efficiently and conveniently identifies the knowledge point boundary using the local minimum of the similarity calculated by ROM. The recursive decision tree algorithm ensures that the time lengths of all knowledge point micro-videos are quite similar. Through comprehensive evaluation indicators such as mAP, Precision, Recall, the model performance is effectively optimized, and it performs excellently in multiple test scenarios, especially in high-resolution images and complex backgrounds. Brief Description of the Drawings

[0037] The drawings described herein are provided to further understand the present application and form a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application.

[0038] Figure 1 It is a schematic diagram of the process structure of the present invention;

[0039] Figure 2 It is a schematic diagram of slide extraction of the present invention;

[0040] Figure 3 It is a schematic diagram of the process of title detection and recognition of the present invention;

[0041] Figure 4 It is a schematic diagram of the process of video segmentation of the present invention;

[0042] Figure 5 It is a schematic diagram of the EfficientViT module used in the present invention;

[0043] Figure 6 It is a schematic diagram of the SE module used in the present invention;

[0044] Figure 7 It is a schematic diagram of video entity recognition in the embodiment of the present invention;

[0045] Figure 8 It is a schematic diagram of video segmentation in the embodiment of the present invention. Detailed Description of the Invention

[0046] Lecture videos have become the most popular learning resources for students. They are usually organized by chapters and have a duration of about 50 minutes or longer, which makes it inconvenient to quickly locate specific knowledge points. Therefore, the work of generating multi-level catalogs for videos has emerged. These works mainly focus on video entity recognition and chapter and section catalog recognition, and the smallest catalog level is the third-level title. The method of marking videos with catalogs may result in larger or smaller abnormal segments, which is still not sufficient for users to quickly locate specific knowledge points.

[0047] To solve this problem, a segmentation task based on semantic and temporal constraints is proposed, focusing on knowledge boundary recognition (KBR), which involves splitting large knowledge points into smaller segments and merging smaller knowledge points. Here, the KBR problem is transformed into a video content relevance (VCR) problem. Therefore, KBR is the minimum point of VCR.

[0048] However, calculating the Video Content Relevance (VCR) is not an easy task. The knowledge points in teaching can usually be distinguished under the second-level headings, and they can also be summarized from the paragraphs. Therefore, it is necessary to identify the first-level and second-level headings, as well as the video entities at the paragraph level. Currently, the best model for identifying fine-grained video entities such as various levels of headings and paragraphs is ResNet50, with an accuracy of about 86%. The accuracy is still relatively low for knowledge point segmentation. To significantly improve the accuracy, this application adopts an improved YOLOv8 model.

[0049] Some knowledge points may span multiple PPT slides, while others may cover 2-3 knowledge points on a single slide. This feature poses a challenge to the multi-scale feature learning ability of the YOLO model. For this reason, the optimal EfficientViT network and SE attention mechanism are adopted to enhance YOLOv8.

[0050] The specific work process is as follows:

[0051] For the extraction of PPT pages, first, frame the teaching video frame by frame per second, use algorithms such as Average Hash, Difference Hash, Perceptual Hash, and Histogram Similarity to measure the similarity between pictures, retain the first picture in each group of duplicate pictures, and remove the remaining pictures. Through the above operations, PPT page pictures are successfully obtained from the video, and the redundancy rate is kept at a low level.

[0052] Title detection and recognition: First, use the deduplicated PPT page pictures to construct a title detection task dataset, which is used to train the first-level title detection model after manual annotation; the pictures after cutting off the first-level titles detected by the model are re-annotated for training the second-level title detection model. Here, the YOLOv8 with better object detection performance is used to train the title detection model. Use the trained title detection model to detect the titles in the video and store them in a file in the form of "appearance time: title" in the order of their first appearance time.

[0053] Video segmentation: Use the ROM semantic relevance model to calculate the relevance scores between adjacent titles, and take the local minimum points on the score curve as the preliminary video segmentation positions. Set a time window of [θ, η]. For the video blocks that need to be merged, compare the relevance scores between the titles in this video block and the titles in its adjacent video blocks, and merge the video block to the video block where the title with a higher relevance score is located. For the video blocks that need to be further split, first split them using the first-level titles. If the requirements cannot be met, introduce the second-level titles. If there are still video blocks that cannot be split later, introduce the text content to continue splitting until the requirements are met.

[0054] AsFigure 1 - Figure 6 As shown, it is the overall process of teaching video segmentation. The flowchart mainly consists of three parts: PPT page extraction, title detection and recognition, and video adaptive segmentation. The specific process: video frame extraction → duplicate removal → detection and recognition of titles and text → semantic relevance chunking → time balance chunking → chunking results.

[0055] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0056] Embodiment:

[0057] In the proposed method, the video segmentation method with time and semantic constraints includes three modules: slide extraction, knowledge point boundary recognition (KBR), and segmentation point recognition (STC). Among them, STC considers semantic and time factors, and the architecture is as Figure 1 shown.

[0058] For the extraction of slides, in order to obtain each title in the video and its first appearance time, the present invention needs to extract frames from the video. Since too large a frame extraction interval may cause a frame when a certain title first appears to be exactly not extracted, resulting in an error between the obtained first appearance time of the title and the actual one, thus affecting the accuracy of the subsequent chunking results. Therefore, frame extraction is performed on the video frame by frame per second. However, this will result in a large number of pictures in the frame extraction results and a high repetition rate. So, a duplicate removal operation is performed on the video frame extraction results. Here, average Hash (aHash), difference Hash (dHash), perceptual Hash (pHash), and RGB histogram similarity (RGBh) algorithms are used to measure the similarity between pictures, and the following formula is used to determine whether pictures are repeated:

[0059] A = Θ(aHash pic1 , aHash pic2 )

[0060] D = Θ(dHash pic1 , dHash pic2 )

[0061] P = Θ(pHash pic1 , pHash pic2 )

[0062]

[0063] Among them, it is used to calculate the number of different bits in two hash values; flag being 1 indicates that pic1 and pic2 are duplicates, and flag being 0 indicates that pic1 and pic2 are not duplicates. Among them, Θ represents the difference between the hash values of different titles, flag represents the flag bit, a flag bit of 1 indicates that the two images are similar, and a flag bit of 0 indicates that the two images are not similar. A represents the hash value calculated by the Average Hash algorithm; D represents the hash value calculated by the Difference Hash algorithm; P represents the hash value calculated by the Perceptual Hash algorithm; avgHash represents the average value of the three hash values; s1 represents the upper limit of the preset threshold, and s2 represents the lower limit of the preset threshold. s.t.A represents the similarity metric between the average hash values of two images. pic1 and pic2 represent Image 1 and Image 2;

[0064] The idea during deduplication is to keep the first image in each group of duplicate images and remove the rest. Therefore, starting from the first image that is determined to be dissimilar to the previous images, calculate the similarity between this image and the subsequent images, and delete the images that appear after this image and are determined to be duplicates of this image. Through the above operations, PPT page images are successfully obtained from the video, and the redundancy rate in the images is maintained at a relatively low level.

[0065] For title detection and recognition, since different PPTs have different page design styles or layouts, the titles may appear in different sizes and positions. It is possible that the title, text, and images in some pictures are more concentrated, while in some others, they may span a larger area. Especially in the case where the title spans multiple areas or is covered by the background or image elements, the convolutional operation of the backbone network Darknet-53 of the YOLOv8 model generally has a general effect in perceiving the title position globally, and Darknet-53 cannot well filter out important local features to accurately separate the title from the content areas such as text, images, and background. In addition, the PPT background is diverse and may have images, textures, or other complex elements, and the YOLOv8 model is prone to being interfered by these irrelevant information, thus affecting the accuracy of title detection. Therefore, in the present invention, the YOLOv8 backbone network is replaced by EfficientViT to balance the perception of global information and local information. Secondly, the SE attention mechanism is introduced to reduce the attention to interference information by considering the correlation relationship between feature channels. In this application, the improved model is named the YOLOv8-ES model.

[0066] For the EfficientViT Backbone, this module uses a multi-scale linear attention mechanism to perceive global and multi-scale information. Using ReLU as the similarity function, for a given The softmax attention formula is:

[0067]

[0068] This formula can be simplified to

[0069]

[0070] Q represents the Query matrix, which is usually the features extracted from the input data; K represents the Key matrix, also the features extracted from the input data, and V represents the Value matrix, also the features extracted from the input data, which is used for the final weighted summation; ReLU represents applying the ReLU activation function to the elements in the matrix, that is, ReLU(x) = max(0, x); ∑ represents the summation symbol, indicating the summation of all elements; the Softmax function is used to normalize the attention scores into a probability distribution.

[0071] During each query, the already calculated and can be used to reduce redundant operations. While obtaining the global receptive field through ReLU linear attention, a depth convolution is inserted in each FFN layer to focus on local information. In addition, depthwise separable convolutions are used to obtain multi-scale tokens from nearby Q / K / V, and group convolutions are used to reduce the computational cost. In this model, EfficientViT is used to replace Darknet-53 in the backbone part of yolov8 to balance the model's attention to global and local information, and multi-scale information is added to enhance the model's multi-scale learning ability.

[0072] The SE attention mechanism uses squeeze and excitation operations to increase the dependency information between channels such as Figure 5 . The feature map X ∈ R(H'×W'×C') input to the SE module is first mapped to U ∈ R(H×W×C), and then the global average pooling operation is used to concentrate the information in the feature map U onto a 1×1×C vector, denoted as z c :[[]]

[0073]

[0074] where F sq represents the operation function for performing global average pooling on the input u t , u t represents the input feature map, with dimensions H×W, where H is the height of the feature map and W is the width of the feature map, i and j are the row index and column index on the feature map respectively, and u t (i, j) represents the feature map u tThe pixel value at (i,j), and then this 1×1×C vector is compressed to a 1×1×C / R dimensional vector through a fully connected layer. After ReLU activation and a fully connected layer, it is restored to a 1×1×C vector. After Sigmoid activation, the weight information s between channels in the feature map U is obtained, where s = F ex (z, W) = σ(W2δ(W1z)) multiplies the vector representing the channel weights of the feature map U with the feature map U to obtain a channel-weighted feature map

[0075] In this model, the SE attention mechanism is added after the EfficientViT module to enhance the model's attention to effective information.

[0076] Use the improved YOLOv8-ES model to detect the images after frame extraction and duplicate removal of the video, locate and intercept the titles in the images, record the first appearance time of each title, and then use PaddleOCR to convert the titles into text for subsequent use.

[0077] Store the recognized first-level titles in the order of their first appearance time in the form of "appearance time: first-level title" into a file. Starting from the first title in this file, use the ROM semantic correlation model to calculate the correlation scores between adjacent titles in sequence, and take the local minimum points on the score curve as the preliminary video segmentation positions.

[0078] title list = [ti0, ti1, ti2, ti n-2 , ti n-1

[0079] This is a list of first-level titles arranged in the order of appearance time

[0080] E list = [E(ti0, ti1), E(ti1, ti2),..., E(ti n-2 , ti n-1 )

[0081] This is a list of correlation scores between adjacent titles. E(ti i , ti i+1 ) represents the correlation score between the i-th title and the (i + 1)-th title;

[0082] min indices = {i | (Δ 2 E list ) i > 0}

[0083] This is a set of index sets, indicating that in E list ​Indices of all local minimum points found here. Here, Δ 2 E list represents the second-order difference used to detect local minima;

[0084] min values ={E list [i] | i ∈ min indices}

[0085] min points ={(i, E list [i - 1]) | i ∈ min indices}

[0086] min indices is the set of local minimum indices, and min values represents the set of local minimum correlation values.

[0087] To achieve a video block length Ω with similar semantics, where "similar" is defined as the time interval [θ, η], we use a recursive decision tree algorithm. First, we use L1 headings to segment knowledge points. If we encounter a block with Ω > η, we introduce a second-level heading. If further splitting is required, we introduce paragraph content for continued splitting. If we encounter a block with Ω < θ, we need to merge adjacent blocks. We alternate between merging and splitting operations until Ω ∈ [θ, η]. When merging adjacent blocks, we prefer to merge blocks with higher similarity. The splitting strategy includes finding local minima in the sub-video entities of the block to be split. If the same block requires secondary splitting, we use an equal division strategy, that is

[0088] The operations selected for the initially segmented ones are as follows:

[0089]

[0090] where Ω represents the segmented video block. If the duration is less than θ, the video block is merged with the adjacent video block. If the duration is greater than γ, it is further split. For video blocks that need to be merged, we compare the correlation scores between the headings in this video block and the headings in its adjacent video block, and merge the video block onto the video block where the heading with a higher correlation score is located. The specific formula is as follows:

[0091]

[0092] For video chunks that need to be further split, first try to split them through the first-level headings in the video chunk: traverse the first-level headings in the video chunk one by one, find the first-level headings that can split the video into sub-chunks of [100s, 300s], and then determine whether the video chunk after splitting into sub-chunks needs to be further split. If so, continue to split according to the above splitting strategy until the requirements are met or it is still necessary to continue splitting after traversing. If there are video chunks that need to be further split, it means that the video chunk cannot be reasonably split based on the first-level headings alone, so the second-level headings need to be introduced. Crop the area above the first-level heading and its higher levels in the pictures within the video chunk to be split to avoid the influence of the first-level heading on the second-level heading detection model.

[0093] Then extract the detected second-level headings and their first appearance times and insert them into the headings contained in the original video chunk in chronological order. Continue to use the above splitting strategy to split the video chunk. For video chunks that still cannot be split using the second-level headings, introduce the body content to continue splitting. The body content is the remaining part of the picture after removing the first-level and second-level headings. Use PaddleOCR to convert the body content into text.

[0094] Judge whether the content is the same by comparing the number of different characters between the recognized texts at different time points. Considering the error of OCR, even for the same content, there may still be a small number of different characters in the finally recognized texts. Therefore, count the number of different characters in the body content of adjacent pictures. If a certain value is significantly greater than its adjacent value, it means that the body content has changed. Therefore, record the body content of this picture and its appearance time, insert it into the headings contained in the original video chunk in chronological order, and continue to use the above splitting strategy to split the video chunk; if the number of different characters in the body content of adjacent pictures is always very small after counting and there is no value significantly greater than other values, it means that the video is always talking about the same part of the content. Since its duration is too long, evenly divide this video. The calculation strategy for the number of split video chunks is as follows:

[0095] It is worth mentioning that although the time window set in this application is [100s, 300s], there will be some video chunks longer than 300 seconds. The reason is that the system will automatically merge chunks shorter than 100 seconds into other chunks, and after merging, there may be chunks with a total duration of 300 to 400 seconds. Considering that these chunks do not exceed the time window by too much and splitting them again may cause chunk chaos, they are directly retained.

[0096] Embodiment

[0097] There are mainly two datasets for the video chunking task in this embodiment, namely the discrete mathematics teaching video dataset and the CCF academic conference report video dataset. Although the two datasets have different topic types, they are both explained based on PPT. The brief feature information of the two datasets is shown in Table 1, and the video duration distribution is as Figure 7 .

[0098] Table 1 Dataset Comparison

[0099]

[0100] For the dataset, 29 sets of 1,685 discrete mathematics lecture videos were crawled from the network. After slide extraction, 300,000 images extracted from 148 randomly selected videos were used to evaluate the slide extraction algorithm. From the de-duplicated images, 11,000 images with no obvious feature repetition were selected and labeled for training and validating the first-level title detection model, and the division ratio of the training set to the validation set was 9915:1085.

[0101] Frames were extracted from the videos in the discrete mathematics teaching video dataset and the CCF academic conference report video dataset, and the above de-duplication algorithm was applied for de-duplication. The de-duplicated pictures were divided into two categories according to PPT features: regular (with a table of contents) and messy (without a table of contents). Then, multiple datasets of the following scales were constructed. The pictures in each dataset were labeled using the labelimg tool and divided into a training set and a validation set according to a ratio of 9:1 for training the first-level title detection model.

[0102] a) 5500 regular data

[0103] b) 5500 messy data

[0104] c) 2750 regular data + 2750 messy data

[0105] d) 11000 regular data

[0106] e) 11000 messy data

[0107] f) 5500 regular data + 5500 messy data

[0108] After removing the first-level titles from the pictures in the first-level title detection model dataset, 6086 pictures containing second-level titles were selected and labeled for training the second-level title detection model, training set:validation set = 5477:609.

[0109] 141 videos were randomly selected from the 2 video datasets, and the first-level titles, second-level titles, and body contents (a total of 2359 items) in the videos were extracted, labeled, and used to evaluate the semantic relevance chunking method.

[0110] 1000 video chunking results were randomly selected and labeled from two video datasets to evaluate the performance of the video adaptive chunking method in terms of temporal balance and accuracy.

[0111] For the evaluation metrics, let A denote the number of images obtained through frame extraction, α denote the number of images removed by the deduplication algorithm, B denote the number of groups of duplicate images after frame extraction, and β denote the number of the first images in each group of duplicate images retained by the deduplication algorithm. The performance of image deduplication is evaluated by recall (R), deduplication rate (DR), redundancy rate (RR), and average similarity (AS), and the formulas are as follows:

[0112]

[0113] Precision (P), recall (R), mean average precision (mAP), accuracy, and F1 score are used to evaluate the performance of the first-level and second-level title detection models and the semantic relevance segmentation method. The video segmentation performance is evaluated by average segmentation duration (ACD), extreme duration ratio (EDR), intersection over union (IOU), and the accuracy of the segmentation results.

[0114] 1) Analysis of image deduplication performance: In this study, histogram similarity was used as the similarity threshold s2, and s2 was set to 0.5, 0.6, 0.7, 0.8, and 0.9. The results are shown in Table 1. When s2 = 0.9, the recall is close to 100%. Therefore, the similarity threshold of the deduplication algorithm in this study was determined to be 0.9. Table 2 shows the deduplication results using this algorithm on datasets of different sizes, indicating that the deduplication algorithm has excellent performance.

[0115] Table 1: Influence of different similarity thresholds s2 on deduplication results

[0116]

[0117] Table 2: Deduplication results with similarity threshold s2 = 0.9

[0118] Data volume 10000 50000 100000 200000 300000 Repeated group 302 798 4539 7621 11995 DR 89.1％ 88.6％ 88.7％ 88.3％ 88.1％ RR 10.5％ 11.4％ 11.3％ 11.7％ 11.9％ AS 93.71％ 92.91％ 94.82％ 91.73％ 94.74％

[0119] 2) Analysis of the results of the title recognition model: Parameters of EfficientViT. Comparative experiments were conducted on six variants M0 - M5 of EfficientViT, and the results are shown in Table 3. The R values of all combinations are not lower than those before improvement. YOLOv8 - ES (M5) demonstrated the best overall performance.

[0120] Table 3 Influence of EfficientViT with different parameters on the performance of the first - level title detection model

[0121]

[0122] In the ablation experiment, the experimental results of YOLOv8, YOLOv8 + E (EfficientViT), YOLOv8 + S (SE), and YOLOv8 + E + S (YOLOv8 - ES) are shown in Table 4.

[0123] Table 4: Influence of different modules on the performance of the first - level title model

[0124]

[0125] After adding two modules, all indicators have improved compared to the original model, enhancing the performance of the model. In addition, the overall performance of the combined model is better than that of the model with only one module added.

[0126] Evaluation of the second - level title model. For the second - level title detection model, since it already showed good performance before modification, no improvement was made. Table 5 shows the evaluation indicators of this model, indicating that the model can detect all categories as comprehensively as possible and has high accuracy.

[0127] Table 5: Evaluation results of the second - level title detection model

[0128]

[0129] 3) Semantic - based segmentation: The results of semantic segmentation are shown in Table 6. It can be observed that using the minimum point on the correlation score curve for segmentation can achieve high accuracy, effectively meeting the requirements of automatic segmentation.

[0130] Table 6 Correlation analysis

[0131]

[0132] 4) Video chunking: The time window [θ, η] is set to [100s, 300s], and the evaluation results of the video segmentation algorithm are shown in Table 7:

[0133] Table 7: Video chunking results

[0134] IoU Accuracy ACD EDR 94.40％ 90.20％ 192.94s 1.15％

[0135] The accuracy rate of the video segmentation method exceeds 90%, and only 1.15% of the segments exceed the set time window. This indicates that the method can ensure the semantic consistency within each video segment and maintain a close time balance, effectively achieving these two goals.

[0136] Figure 7 and Figure 8 shows an example of video entity recognition and video segmentation results. In Figure 7 , the L1 headings, L2 headings, and paragraphs are correctly recognized with a high confidence level. In Figure 8 , the finally segmented videos are all between 100s and 300s, and the content in each cut is about the same knowledge point, meeting the expectations.

[0137] This application proposes a knowledge point-level lecture video segmentation method based on semantic and temporal constraints. A four-in-one slide extraction algorithm is designed, an enhanced KBR method combining YOLO++ and MROM technologies is proposed, and a recursive decision tree algorithm is used to achieve the adaptive balance of semantics and time. In the experiment, the method achieved satisfactory results, where the accuracy rate of YOLO++ in video entity recognition reached 94.7%, significantly higher than the existing models. The segmentation accuracy rate with dual semantic and temporal constraints also reached 90.2%.

[0138] It should be noted that in the present invention, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device.

[0139] The above embodiments are only illustrative examples of the present invention and do not constitute a limitation on the protection scope of the present invention. Any design identical or similar to the present invention falls within the protection scope of the present invention.

Claims

1. A method for generating teaching micro-videos based on semantic and time constraints, characterized in that, Including the following steps: Frame the teaching video frame by frame and use multiple hash algorithms to remove duplicates, obtaining multiple non-repeating slide images; Replace the Darknet-53 backbone network in YOLOv8 with EfficientViT and introduce the SE attention mechanism to construct an improved YOLOv8 model. Use the improved YOLOv8 model to identify and extract slides from each de-duplicated frame of the slide image, obtaining the title of the slide image and the timestamp when the title first appears; Use a semantic correlation model to associate the titles of the slide images, and obtain the local minimum of the similarity between titles as the knowledge point boundary; Use the recursive decision tree algorithm, combined with the knowledge point boundary and the set time window, to segment the teaching video to obtain the segmentation result. Based on the semantic correlation between the segmentation results, merge the segmentation results to form teaching micro-videos corresponding to knowledge points.

2. The method for generating teaching micro-videos based on semantic and time constraints according to claim 1, wherein, After obtaining the segmentation result, for the nodes not within the set time window, use the equal division strategy to perform secondary segmentation on the segmentation result to obtain the secondary segmentation result. Based on the semantic correlation, merge the secondary segmentation result to form teaching micro-videos corresponding to knowledge points.

3. The teaching micro-video generation method based on semantic and time constraints according to claim 1, characterized in that The obtaining of the multiple non-repeating slide images includes the following steps: Divide the teaching video into a series of consecutive image frames; preprocess each frame of the image to obtain each processed frame of the image; Use the Average Hash algorithm, Difference Hash algorithm, Perceptual Hash algorithm, and Histogram Similarity algorithm to obtain the similarity between each processed frame of the image. Use redundancy to represent the similarity function, retain the first image in each group of duplicate images, and remove redundant frames to obtain multiple non-repeating slide images.

4. The teaching micro-video generation method based on semantic and time constraints according to claim 1, characterized in that, The using of the improved YOLOv8 model to identify and extract slides from each de-duplicated frame of the slide image, obtaining the title of the slide image and the timestamp when the title first appears, includes: Use the improved YOLOv8 model to perform object detection on the set of de-duplicated slide images, locate and intercept the regions of the first-level titles and second-level titles, convert the regions of the first-level titles and second-level titles into text, and record the timestamp when each title first appears in the video to obtain the title of the slide image and the timestamp when the title first appears.

5. The method for generating teaching micro-videos based on semantic and temporal constraints according to claim 1, characterized in that, The obtaining of the knowledge point boundary includes the following steps: Select the semantic similarity model as ROM; Preprocess the title text of each slide to obtain the preprocessed title text; Convert the preprocessed title text into a semantic vector representation; Associate each slide image with its corresponding title text, and use the timestamp of the slide as an identifier; Use the ROM semantic similarity model to sequentially obtain the similarity scores between the semantic vectors of adjacent slide titles, and use the local minimum points on the score curve as the preliminary video segmentation positions; Until the local minimum of the similarity between titles is obtained as the knowledge point boundary.

6. The teaching micro-video generation method based on semantic and temporal constraints according to claim 2, wherein, Using the recursive decision tree algorithm, combining the knowledge point boundaries and the set time window, segment the teaching video to obtain the segmentation result, including the following steps: Set the time window [θ, η], where θ is the minimum segmentation duration and η is the maximum segmentation duration. Treat the teaching video as a whole as the initial node; Select a knowledge point boundary as the segmentation point and divide the initial node into two child nodes; For each child node, when the duration of the set window exceeds the maximum segmentation duration η, segment it; When a node contains multiple knowledge points, segment the node containing multiple knowledge points until the duration of the set window of the node is less than the maximum segmentation duration and there is one knowledge point in the node to obtain the segmentation result.

7. The teaching micro-video generation method based on semantic and time constraints according to claim 6, characterized in that, The secondary segmentation of the segmentation result using the equal division strategy includes the following steps: According to the segmentation result, obtain the remaining part of the slide image after removing the first-level and second-level headings as the body text part; Convert the image of the body text part into text, and count the number of character-level differences in the text of the body text parts of adjacent slides to obtain the character difference value; If the character difference value of the slide exceeds the average value of adjacent slides, it is determined that there is a substantial change in the body text content. Record this time point and the corresponding body text content as potential knowledge point boundaries, and use the recursive decision algorithm combined with the time window constraint to perform secondary segmentation on the nodes not within the time window [θ, η]; If the character difference value of adjacent slides is lower than the threshold, it is determined that the content of this video block belongs to the same knowledge point, but its duration exceeds the preset maximum time window. Divide this video block into several sub-blocks so that the duration of each sub-block falls within the allowed time window; if the duration of the sub-block after equal division is less than the minimum time window, merge this sub-block with the adjacent sub-block to make the segmentation result meet the time window constraint and obtain the secondary segmentation result.

8. A generation system for a teaching micro-video generation method based on semantic and time constraints, characterized in that, Including: A teaching video splitting module for extracting frames from the teaching video frame by frame and using multiple hashing algorithms for duplicate removal processing to obtain non-repeating slide images for each frame after duplicate removal; A split image processing module for replacing the Darknet-53 backbone network in YOLOv8 with EfficientViT and introducing the SE attention mechanism to construct an improved YOLOv8 model. Use the improved YOLOv8 model to identify and extract slides from the non-repeating slide images for each frame after duplicate removal to obtain the title of the slide image and the timestamp when the title first appears; associate the titles of the slide images where they are located by combining semantics with the title, and obtain the local minimum value of the similarity between the titles as the knowledge point boundary; A micro-video synthesis module for using the recursive decision tree algorithm, combining the knowledge point boundaries and the set time window, segmenting the teaching video to obtain the segmentation result, and merging the segmentation result based on semantic relevance to form teaching micro-videos corresponding to knowledge points.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 above is implemented.

10. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method according to any one of claims 1 to 7 above.

Images