Method, device and storage medium for automatic positioning of educational videos

Abstract

The present invention provides a method, device and storage medium for automatic positioning of educational videos. The method includes: obtaining uploaded educational videos and educational courseware; extracting video stream features of the educational video based on a deep network model to generate a video key frame sequence; extracting a text sequence in each key frame image based on an OCR method to generate a key frame text sequence; extracting the structured information of the educational courseware and outputting the content of each page of the courseware in text form; applying a courseware-video positioning algorithm to the courseware text sequence and the key frame text sequence to automatically locate the video explanation position corresponding to each page of the educational courseware. The present invention realizes automatic positioning from educational courseware to video. On the one hand, it can provide resource managers of the educational resource platform with an accurate and intelligent video positioning function, reducing the cost of manual labeling; on the other hand, users can reduce the blindness and randomness of the retrieval process during use, thereby improving the flexibility and learning efficiency of online learning, thereby enhancing the user experience.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an automatic positioning method, device and storage medium for educational videos. Background Art

[0002] In blended learning, multimedia courseware, course videos, and academic lectures are becoming increasingly common, and these educational resources are widely used on educational resource management platforms. Typically, after a course, instructors upload recorded educational videos and accompanying courseware to the educational resource management platform for learners to watch and study independently after class. However, existing platforms have the following major shortcomings in educational resource management:

[0003] (1) Videos are long and contain many knowledge points, making it inconvenient to conduct targeted review of difficult knowledge points. Generally speaking, learners are basically in a passive learning state when learning video courses, with a heavy cognitive load. They need to watch the video repeatedly to understand the more difficult parts of the knowledge. Videos are dynamic media with strong controllability, making it difficult for learners to accurately locate the key points of learning through the video screen. Excessively long videos increase the blindness of the video positioning process, making it inconvenient for fragmented review.

[0004] (2) The resources between educational videos and supporting courseware are independent of each other, and there is no mapping relationship between the two. When learners are reviewing courseware and do not understand the content of a certain page of courseware, they want to watch the explanation of that page in the video. However, most existing platforms do not provide a jump function from courseware to video. Learners can only manually find the corresponding location of the video, which greatly reduces learning efficiency and user experience.

[0005] (3) Manual video annotation is time-consuming and labor-intensive, and is prone to annotation errors. Although some platforms provide a jump function from courseware to video, most of them require manual annotation of the mapping relationship between each page of courseware and the video timestamp. With the continuous generation of massive educational resources, manual video annotation will consume a lot of manpower costs, and the process is cumbersome and prone to annotation errors. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the present invention provides a method, device and storage medium for automatic positioning of educational videos, which aims to quickly jump to the corresponding video explanation position through the educational courseware content.

[0007] The technical means adopted in the present invention are as follows:

[0008] An automatic positioning method for educational videos, comprising the following steps:

[0009] S1. Obtaining uploaded educational resources, wherein the educational resources include educational videos and educational courseware. The educational videos are course videos recorded during teaching, academic lectures, academic conferences, and scientific research training. The educational videos include courseware slideshow content. The educational courseware is courseware that complements the content explained in the educational videos.

[0010] S2. Extracting video stream features of the educational video based on a deep network model to generate a video key frame sequence, wherein the video key frame sequence includes non-repeated video images;

[0011] S3, extracting the text sequence in each key frame image based on the optical character recognition method to generate a key frame text sequence;

[0012] S4, extracting the structured information of the educational courseware, and outputting the content of each page of the courseware in text form to generate a courseware text sequence;

[0013] S5. Using a courseware-video positioning algorithm on the courseware text sequence and the key frame text sequence, automatically locate the video explanation position corresponding to each page of the educational courseware.

[0014] Furthermore, S2, extracting video stream features of the educational video based on a deep network model to generate a video key frame sequence, including:

[0015] S201, performing frame processing on the educational video to generate frame images with time stamps;

[0016] S202, extracting image features of the framed image through a convolutional neural network;

[0017] S203, defining the first framed image as a key frame;

[0018] S204: Calculate similarity of features of adjacent frame images based on the image features. If the similarity is less than a preset threshold, use the next frame as a key frame, and use the time point of the key frame in the video as the timestamp of the key frame. Otherwise, move to the next image and continue calculating the similarity.

[0019] S205 , output the key frame image sequence number frame_id and the key frame timestamp frame_ts.

[0020] Furthermore, the convolutional neural network is one of VGG, GoogleNet, ResNet, DenseNet, MobileNet and ShuffleNet.

[0021] Furthermore, S3 extracts the text sequence from each key frame image based on an optical character recognition method, including:

[0022] S301, pre-processing the key frame image;

[0023] S302, performing text detection on the pre-processed key frame image, and returning the position coordinates of the line where the text is located in the form of a rectangular frame;

[0024] S303, performing text recognition based on the text detection, and converting the rectangular frame area into text;

[0025] S304: Sort all recognized texts in ascending order according to the coordinates of their upper left corners. The text sequence of the i-th key frame image is represented as frame_kw[i]=[kw1, kw2, ..., kw n ], where i = {1, 2, ..., K}, K represents the number of key frames, n represents the number of texts recognized in the current key frame image, kw n Indicates the nth text recognized in the previous keyframe image.

[0026] Furthermore, text detection is performed on the preprocessed key frame image, including text detection through one of Faster R-CNN, FCN, RRPN, TextBoxes, CTPN and SegLink.

[0027] Furthermore, S4 extracts the structured information of the educational courseware and outputs the content of each page of the courseware in text form, including:

[0028] S401, reading an educational courseware document, calling a document analyzer to parse the educational courseware document, and returning the types and text position coordinates of all objects contained in each page of the courseware, wherein the types of the objects include text, images, tables, and curves, and the text position coordinates are represented by the coordinates of a rectangular box of a text object area;

[0029] S402, using an optical character recognition method to perform text recognition on non-text objects in the educational courseware, and directly reading the text content of text objects in the educational courseware;

[0030] S403, sort all recognized texts in ascending order according to the coordinates of their upper left corners, and the text sequence of the j-th page of courseware is expressed as ppt_kw[j]=[kw1, kw2, ..., kw m ], where j = {1, 2, ..., P}, P represents the total number of pages of the courseware, m represents the number of texts recognized in the courseware on this page, kw m Indicates the mth text recognized by the courseware on this page.

[0031] Furthermore, S5, a courseware-video positioning algorithm is used on the courseware text sequence and the key frame text sequence to automatically locate the video explanation position corresponding to each page of the educational courseware, including:

[0032] S501, obtaining a key frame number list corresponding to the text sequence, wherein the text sequence is frame_kw[i]=[kw1, kw2, ..., kw n ], where i = {1, 2, ..., K}, K represents the number of key frames, n represents the number of texts recognized in the current key frame image, kw n Indicates the nth text recognized by the previous key frame image, and the key frame number list corresponding to the text sequence is frame_numlist[i]=[1, 2, ..., n];

[0033] S502, for each key frame image text sequence frame_kw[i], traverse all educational courseware text sequences ppt_kw[j], j = {1, 2, ..., P}, and perform string fuzzy matching through the text one by one. If the similarity is greater than the set threshold, output the sequence number of the element corresponding to the key frame image sequence number frame_id[i], otherwise output 0, and get the courseware label list for key frame i page j as follows Where ppt_kw[j]=[kw1,kw2,...,kw m ], represents the text sequence of the jth page of the courseware, j = {1, 2, ..., P}, P represents the total number of pages of the courseware, m represents the number of texts recognized in the courseware on this page, kw m Indicates the mth text recognized by the courseware on this page;

[0034] S503. List of courseware labels After processing, when When there are two or more consecutive 0s in the sequence, only one 0 in the sequence is retained to obtain a new courseware number list For each frame_numlist[i], traverse all new Calculate the similarity between the two and return the courseware with the maximum similarity as the courseware number matched by key frame i;

[0035] S504, traverse each key frame image, repeat S503, obtain the number of courseware pages that all key frame images match, sort the key frame groups in ascending order by courseware page number, and return the courseware labels that all key frame images match;

[0036] S505, post-processing the courseware labels matched to all key frame images to find the longest increasing subsequence of the sorted key frame sequence number list;

[0037] S506. After sorting by the longest ascending subsequence, only the key frame with the smallest sequence number in each group is retained as the key frame image matching each page of courseware, and the timestamp of the key frame image is used as the video start time for locating the page of courseware.

[0038] The present invention also discloses an automatic positioning device for educational videos, comprising:

[0039] An educational resource upload module, which is used to obtain uploaded educational resources, including educational videos and educational courseware. The educational videos are course videos recorded during teaching, academic lectures, academic conferences, and scientific research training. The educational videos include courseware slideshow content. The educational courseware is courseware that matches the content explained in the educational videos.

[0040] A video key frame generation module, which is used to extract video stream features of the educational video based on a deep network model to generate a video key frame sequence, wherein the video key frame sequence includes non-repeated video images;

[0041] A key frame text recognition module is used to extract the text sequence in each key frame image based on the optical character recognition method to generate a key frame text sequence;

[0042] A courseware structured extraction module is used to extract the structured information of the educational courseware and output the content of each page of the courseware in text form to generate a courseware text sequence;

[0043] The courseware-video positioning module is used to automatically locate the video explanation position corresponding to each page of the educational courseware by using the courseware-video positioning algorithm on the courseware text sequence and the key frame text sequence.

[0044] The present invention also discloses a storage medium, which includes a stored program, wherein when the program is run, any one of the above-mentioned methods for automatically positioning educational videos is executed.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] 1. The present invention proposes an automatic positioning method for educational videos that can replace manual labeling to complete accurate automatic positioning from educational courseware to educational videos, effectively improving the flexibility of online learning and enhancing user efficiency.

[0047] 2. This invention proposes a video keyframe generation technology that significantly reduces the number of images required to convert a video, alleviates the computational burden on subsequent modules, and improves overall operational efficiency. Furthermore, the generated keyframe images exhibit low similarity and are highly representative, serving as an overview of the key content of a video.

[0048] 3. This invention proposes a method for keyframe text recognition and courseware structured extraction, which is an important foundation for the subsequent courseware-video positioning accuracy. By using text recognition technology and structured extraction methods, this invention refines the matching granularity to the comparison between text content, rather than simply measuring pixel-level similarity between images. This effectively avoids the problem of matching failures caused by large image differences between keyframes and courseware due to differences in image resolution, video recording angle, video background, etc.

[0049] 4. The present invention proposes a courseware-video positioning algorithm. In the process of matching each page of courseware with the key frames of the video, multiple factors such as the text position relationship, OCR fault tolerance, and the animation effect of the courseware playback are considered to achieve accurate matching from courseware to video. Specifically, 1) the text position relationship fully integrates text information and text position information. Its matching method is more in line with human experience. Multi-information fusion improves the accuracy of matching by obtaining multi-dimensional information; 2) OCR fault tolerance takes into account the difficulty difference between video key frames and courseware in OCR tasks, relaxes the matching criteria through string fuzzy matching, and improves the matching rate of keyframe text recognition; 3) The animation effect of courseware playback is one of the commonly used projection methods for educational courseware. The method of the present invention can match each page of courseware to the time point when the page of courseware is first played in the video, which is more in line with the user needs of actual usage scenarios.

[0050] 5. The present invention proposes a key frame LIS post-processing method, introduces the idea of dynamic optimization to address the impact of erroneous matching results, and increases the fault tolerance and stability of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0052] Figure 1 This is a flow chart of an automatic positioning method for educational videos according to the present invention.

[0053] Figure 2 This is the courseware-video positioning flow chart of the present invention.

[0054] Figure 3a This is key frame example 1 in the embodiment.

[0055] Figure 3b Schematic diagram of key frame OCR corresponding to key frame example 1 in the embodiment.

[0056] Figure 3cThis is key frame example 2 in the embodiment.

[0057] Figure 3d Schematic diagram of key frame OCR corresponding to key frame example 2 in the embodiment.

[0058] Figure 3e This is key frame example 3 in the embodiment.

[0059] Figure 3f Schematic diagram of key frame OCR corresponding to key frame example 3 in the embodiment.

[0060] Figure 4a This is PPT type courseware resource example 1 in the embodiment.

[0061] Figure 4b This is PPT type courseware resource example 2 in the embodiment.

[0062] Figure 5a This is the PDF type courseware resource example 1 in the embodiment.

[0063] Figure 5b This is PDF type courseware resource example 2 in the embodiment.

[0064] Figure 6 This is an example of the text position relationship in the embodiment.

[0065] Figure 7a This is an example of a key frame in the OCR fault tolerance example in the embodiment.

[0066] Figure 7b This is a courseware example corresponding to the key frame in the OCR fault tolerance example in the embodiment.

[0067] Figure 8 This is an example of the animation effect of the courseware in the embodiment.

[0068] Figure 9 This is an example of the key frame 116 in the embodiment.

[0069] Figure 10 This is an example of page 2 of the courseware in the embodiment.

[0070] Figure 11 This is an example on page 5 of the courseware in the embodiment.

[0071] Figure 12a This is visualization example 1 in the embodiment.

[0072] Figure 12b This is visualization example 2 in the embodiment.

[0073] Figure 12c This is visualization example 3 in the embodiment.

[0074] Figure 12dThis is visualization example 4 in the embodiment.

[0075] Figure 13 The figure is a structural diagram of an automatic positioning device for educational videos according to the present invention. DETAILED DESCRIPTION

[0076] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0077] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0078] like Figure 1 As shown, the present invention provides an automatic positioning method for educational videos, comprising the following steps:

[0079] S1. Obtain uploaded educational resources, wherein the educational resources include educational videos and educational courseware. The educational videos include courseware slideshow content, and the educational courseware is courseware that matches the content explained in the educational videos.

[0080] Specifically, educational videos primarily refer to course videos recorded during lectures, academic lectures, conferences, scientific research training, and other activities. In particular, these videos may include courseware slideshows, which can be in formats such as MPEG / AVI / MP4 / MKV / FLV. The educational courseware refers to the accompanying courseware used during the video presentation, which can be in formats such as PPT / PDF.

[0081] S2. Extracting video stream features of the educational video based on a deep network model to generate a video key frame sequence, wherein the video key frame sequence includes non-repeated video images.

[0082] Specifically, this step extracts video stream features from the input educational video through a deep network model, generates video key frames in a set of image sequences that are as non-repetitive as possible, and presents a quick overview of the video. It mainly includes the following steps:

[0083] S201, video framing: The input educational video is framed and converted into framed images with time stamps.

[0084] S202: Extract feature vectors from each framed image using a convolutional neural network. The convolutional neural network can be VGG, GoogleNet, ResNet, DenseNet, MobileNet, ShuffleNet, or the like.

[0085] S203. Keyframe detection. Calculate the similarity of features between adjacent framed images. If the similarity is less than a preset threshold, the image is designated as a keyframe, and the corresponding video time point is used as the keyframe's timestamp. Otherwise, move on to the next image and continue calculating similarity. Specifically, define the first framed image as a keyframe.

[0086] S204: Output key frame: Output the video key frame image number (frame_id) and key frame timestamp (frame_ts) obtained in the above steps.

[0087] S3. Extract the text sequence from each key frame image based on the optical character recognition method to generate a key frame text sequence.

[0088] Specifically, this step uses optical character recognition (OCR) technology to extract and return the text sequence in each keyframe image. It mainly includes the following steps:

[0089] S301, image preprocessing. Correct imaging problems of key frame images. Common preprocessing methods include grayscale conversion, distortion correction, blur removal, image enhancement, and light correction.

[0090] S302. Text detection. Detect text areas in the keyframe image and return the coordinates of the text line as a rectangular box. Text detection algorithms can include Faster R-CNN, Fully Convolutional Networks (FCN), Rotation Region Proposal Networks (RRPN), TextBoxes, Connectionist Text Proposal Networks (CTPN), SegLink, and others.

[0091] S303, text recognition. The goal of text recognition is to convert text lines into text based on text detection. Text recognition algorithms can be based on traditional methods such as template matching and sliding windows, or popular deep learning methods such as CRNN and Seq2Seq.

[0092] S304: Output the key frame text sequence. Sort all recognized texts in ascending order according to the coordinates of their upper left corners. The text sequence of the i-th key frame image is represented as frame_kw[i] = [kw1, kw2, ..., kw n ]. Where i = {1, 2, ..., K}, K represents the number of key frames, n represents the number of texts recognized in the image, kw n Indicates the nth text recognized for this image.

[0093] S4. Extract the structured information of the educational courseware, and output the content of each page of the courseware in text form to generate a courseware text sequence.

[0094] Specifically, it is used to extract the courseware structured information and output the content of each page of the courseware in text form. It mainly includes the following steps:

[0095] S401. Read and parse the document. Read the educational courseware document, call the document analyzer to parse the document objects, and return the type and location coordinates of all objects contained in each page of the courseware. Object types include text, images, tables, curves, etc., and text location coordinates are represented by the coordinates of the object area rectangle.

[0096] S402, courseware text recognition: For non-text objects in the courseware, OCR technology is used to perform text recognition; for text objects, the text content is directly read.

[0097] S403, output courseware text sequence. Sort all recognized texts in ascending order according to the coordinates of their upper left corners. The text sequence of the jth page courseware is expressed as ppt_kw[j] = [kw1, kw2, ..., kw m ]. Wherein, j = {1, 2, ..., P}, P represents the total number of pages of the courseware, m represents the number of texts recognized in the courseware on this page, kw m Indicates the mth text recognized by the courseware on this page.

[0098] S5. Using a courseware-video positioning algorithm on the courseware text sequence and the key frame text sequence, automatically locate the video explanation position corresponding to each page of the educational courseware.

[0099] Specifically, this step automatically locates the video explanation position corresponding to each page of the courseware by matching the courseware content with the video keyframe text, and completes the accurate and fast positioning of the courseware to the video. The flow chart is as follows: Figure 2 In particular, this step considers multiple factors such as text position relationship, OCR fault tolerance, and courseware animation effect, and proposes a courseware-video positioning algorithm to achieve accurate matching from courseware to video. Specifically:

[0100] (1) Considering the position relationship of text

[0101] In the process of matching keyframes and courseware, when a keyframe has the same text intersection with multiple pages of courseware, in addition to the similarity between the texts, the positional relationship between the texts should also be considered to improve the accuracy of matching.

[0102] (2) Consider OCR fault tolerance

[0103] Because keyframe images are captured from video frames, they are affected by factors such as the video lens, recording angle, and the teacher's position, resulting in low image resolution and clarity. This makes the keyframe OCR task more difficult, and its text recognition results are inferior to those of the courseware, resulting in a mismatch between the two. To address this issue, the present invention considers OCR fault tolerance and uses a string fuzzy matching method to calculate string similarity. That is, when the similarity between two strings is greater than a set threshold, the two are considered to match each other, thus avoiding the problem of mismatch between keyframes and courseware due to inaccurate OCR recognition.

[0104] (3) Consider the animation effect of courseware playback

[0105] Since courseware may contain animation effects during playback (such as progressive output), the keyframes captured during video recording may only show a portion of the content on that page. However, we want to match each page of courseware to the time when it first plays in the video. Therefore, when locating courseware and video, the animation effects of the courseware playback in the video keyframes should be considered.

[0106] Considering the above three situations, the present invention proposes a courseware-video positioning algorithm to achieve page-by-page mapping of courseware to video key frames, which mainly includes the following steps:

[0107] S501. Label video key frames (taking text position into consideration).

[0108] According to the result returned by the key frame text recognition step S3, the text sequence of the i-th key frame image frame_kw[i]=[kw1, kw2, ..., kw n In order to consider the text position relationship, the key frame number list corresponding to the text sequence frame_kw[i] is expressed as frame_numlist[i]=[1, 2, ..., n]. nIndicates the nth text recognized by the image, and n indicates the number of texts recognized by the image.

[0109] S502. Courseware numbering (taking OCR fault tolerance into consideration).

[0110] According to the result returned by the courseware structured extraction module, the text sequence of the j-th page courseware ppt_kw[j]=[kw1, kw2, ..., kw m ]. For each key frame image text sequence frame_kw[i], traverse all ppt_kw[j], j = {1, 2, ..., P}, considering OCR fault tolerance, perform string fuzzy matching through the text one by one, if the similarity is greater than the set threshold, then output the sequence number of the element corresponding to frame_id[i], otherwise output 0. Based on the above method, the label list for the jth page of key frame i can be expressed as

[0111] S503, post-processing of courseware labeling (taking into account the animation effect of courseware playback).

[0112] In order to consider the animation effect of the courseware, we post-process the courseware label, that is, when When there are two or more consecutive 0s in a sequence, only one 0 in the sequence is retained. This is because The appearance of 0 in the sequence means that the courseware content does not appear in the key frame. However, when there is a courseware animation effect during the video playback (such as line-by-line output), the key frame may just show the beginning of the courseware content. At this time, when the key frame is compared with the complete courseware content, it will cause Outputs many 0s and the match fails. Therefore, through the courseware labeling post-processing method, only One of two or more consecutive 0s in the sequence can effectively match each page of courseware to the video key frame and time point at which the courseware on that page is first played. Indicates that for each frame_numlist[i], traverse all And calculate the similarity between the two, and return the maximum similarity as the courseware matched by key frame i.

[0113] S504. All key frame images match the courseware page numbers.

[0114] By traversing each key frame image and repeating S503 , the number of courseware pages matched by all key frame images can be obtained, and the key frame groups are sorted in ascending order by the courseware page numbers, and the courseware page numbers matched by all key frame images are returned.

[0115] S505: Key frame LIS post-processing.

[0116] Post-process the result returned in step 4 to find the longest increasing subsequence (LIS) of the sorted key frame sequence number list. The purpose is to eliminate the influence of incorrect matching results and increase the fault tolerance and stability of the algorithm.

[0117] S506. Output courseware-video positioning information.

[0118] After sorting by the longest ascending subsequence, only the key frame with the smallest serial number in each group is retained as the key frame image matching each page of courseware, and the timestamp of the key frame image is used as the video start time located in the courseware on that page.

[0119] The scheme and effects of the present invention are further described below through specific application examples:

[0120] This embodiment provides an automatic positioning method for educational videos, including the following steps:

[0121] S1. Obtain uploaded educational resources, which include educational videos and educational courseware.

[0122] S2. Extracting video stream features of the educational video based on a deep network model to generate a video key frame sequence, wherein the video key frame sequence includes non-repeated video images.

[0123] S3, extract the text sequence in each key frame image based on the optical character recognition method to generate a key frame text sequence. In this embodiment, the key frame text recognition module recognizes the key frame image and the key frame OCR result as follows: Figures 3a-3f , Figure 3b 、 Figure 3d and Figure 3f The recognition effects of images containing code blocks, tables, and pictures are shown respectively.

[0124] S4. Extract the structured information of the educational courseware and output the content of each page of the courseware in text form to generate a courseware text sequence. In this embodiment, the courseware structured extraction module extracts the structured information of the courseware resources as shown in FIG4-5. Figure 4a and Figure 4b Displays the structured extraction results of courseware whose resource type is ppt. Figure 5a and Figure 5b This example shows the results of structured extraction for a PDF file. First, the document is parsed and the type of each object in the file is identified. If the object is text, the text content is directly read and output. If the object is non-text, a second OCR is performed on the object, and the text recognition result is output.

[0125] S5. Apply the courseware-video positioning algorithm to the courseware text sequence and the key frame text sequence to automatically locate the video explanation position corresponding to each page of the educational courseware. This embodiment can illustrate the influence of text position relationship, OCR fault tolerance, and courseware playback animation effect on the positioning result during the matching process between the courseware and the video key frames:

[0126] (1) Consider the text position relationship

[0127] For example Figure 6 , the same texts in courseware P1, courseware P2 and key frame K1 are ["HBase data model", "physical view"], but obviously courseware P2 should be matched with key frame K1. Therefore, when performing courseware-video positioning, we not only need to consider the similarity between texts, but also the position relationship between texts.

[0128] (2) Consider OCR fault tolerance

[0129] Perform OCR character recognition on key frame K1 and courseware P1 in Figure 7a and Figure 7b respectively, and the results are shown in Table 1. Through comparison, it is found that in the key frame character recognition, the strings "BETWEEN", "is it empty", "fuzzy matching", "LIKE" are misrecognized as "BEIWEEN", "is it a hundred empty", "fuzzy matching", "LKE". This embodiment considers OCR fault tolerance and uses the fuzzywuzzy string fuzzy matching method process.extractOne, setting the threshold score_cutoff = 70, that is, when the similarity between two strings is greater than 70%, it is considered that the two match. ​​​​​​​​​​​​​​​​​​​​​​​​​ Figure 9 Taking the video key frame image 116 as an example, according to the key frame text recognition module of module three, the key frame image text sequence frame_kw

[116] =[HBase data model, physical view, 00095800] can be obtained, and the corresponding key frame label list frame_numlist

[116] =[1, 2, 3].

[0138] S502, courseware number.

[0139] by Figure 10 、 Figure 11 These are the examples of page 2 and page 5 of the courseware (hereinafter referred to as "courseware 2" and "courseware 5"). According to the module 4 courseware architecture extraction module, we can get the courseware 2 text sequence ppt_kw[2] = [HBase data model, related concepts, concept view, physical view]. Calculate the similarity between each text in the ppt_kw[2] sequence and each text in frame_kw

[116] to find the corresponding label. If "HBase data model" in ppt_kw[2] matches "HBase data model" in frame_kw

[116] , the label is 1; if "related concepts" in ppt_kw[2] do not match elements in frame_kw

[116] , the label is 0; if "physical view" in ppt_kw[2] matches "physical view" in frame_kw

[116] , the label is 3. Therefore, we get the label list of courseware 2 relative to key frame 116.

[0140] Courseware 5 text sequence ppt_kw[5] = [HBase data model, physical view, row key, timestamp, ...]. Calculate the similarity between each text in the ppt_kw[5] sequence and each text in frame_kw

[116] to get the label list of courseware 5 relative to key frame 116.

[0141] S503, post-processing of courseware labeling.

[0142] When there are two or more consecutive 0s in the courseware label list, only one 0 is retained. After the courseware label post-processing, the label list of courseware 2 relative to key frame 116 is List of labels for keyframe 116 in slide 5

[0143] Calculate the similarity between the key frame number list and the courseware number list, and return the courseware page number corresponding to the maximum similarity as the courseware page number matched by the key frame. In this embodiment, the similarity calculation formula of lists list_A and list_B is defined as shown in formula (1).

[0144]

[0145] Among them, edit_distance represents the edit distance between the two lists, and len represents the length of the list. In summary, the similarity between the key frame image 116 and the courseware page number 2 is calculated. Similarity between key frame image 116 and courseware page 5 Therefore, the key frame image 116 is matched to the course page number 5.

[0146] S504. All key frame images match the courseware page numbers.

[0147] Traverse each key frame image, repeat S503, and return the courseware page numbers that all key frame images match, as shown in Table 2. Group the key frame sequence numbers in ascending order according to the courseware page number, and the results are shown in Table 3.

[0148] S505, LIS post-processing.

[0149] According to Table 3, we can get the key frame sequence number list:

[0150] frame_list = [2, 20, 31, 45, 65, 116, 8, 175, 197, 201, 206, 200, 316], find the longest increasing subsequence of frame_list LIS_frame_list = [2, 20, 31, 45, 65, 116, 175, 197, 201, 206, 316], where elements 8 and 200 in the frame_list list are removed. The results after LIS post-processing are shown in Table 4.

[0151] S506. Output courseware-video positioning information.

[0152] Only the smallest keyframe number within each group in Table 4 is retained as the keyframe image corresponding to that page of courseware. Based on the correspondence between the keyframe image numbers and keyframe timestamps output by the video keyframe generation module, a courseware-video location table is obtained, as shown in Table 5. Each row in the table represents the keyframe image number corresponding to the courseware page number and the located video timestamp.

[0153]

[0154] In addition, this embodiment also provides a step of displaying the above positioning results in a visual interface. Figure 12a-12d The following is a visual interface display. The rectangular box on the right indicates the page number of the courseware clicked, and the left side is the jump video position. Figure 12aThe courseware shown has a line-by-line output animation effect during playback, and the present invention can accurately map the time point when the video starts explaining the courseware on this page; Figure 12b As shown, when the resource played in the left video is different from the courseware resource on the right, the text recognition technology and structured extraction method adopted by the present invention are applied to perform text comparison, which can accurately match the video key frames corresponding to the courseware; Figure 12c and Figure 12d As shown, when the courseware content is in plain text form or in a form with pictures, the method of the present invention can obtain correct matching results.

[0155] The present invention also discloses an automatic positioning device for educational videos, such as Figure 13 Shown, including:

[0156] An educational resource upload module, which is used to obtain uploaded educational resources, including educational videos and educational courseware. The educational videos are course videos recorded during teaching, academic lectures, academic conferences, and scientific research training. The educational videos include courseware slideshow content, and the educational courseware is courseware that matches the content explained in the educational videos.

[0157] A video key frame generation module, which is used to extract video stream features of the educational video based on a deep network model to generate a video key frame sequence, wherein the video key frame sequence includes non-repeated video images;

[0158] A key frame text recognition module is used to extract the text sequence in each key frame image based on the optical character recognition method to generate a key frame text sequence;

[0159] A courseware structured extraction module is used to extract the structured information of the educational courseware and output the content of each page of the courseware in text form to generate a courseware text sequence;

[0160] The courseware-video positioning module is used to automatically locate the video explanation position corresponding to each page of the educational courseware by using the courseware-video positioning algorithm on the courseware text sequence and the key frame text sequence.

[0161] As for the embodiment of the present invention, since it corresponds to the above embodiment, the description is relatively simple. For relevant similarities, please refer to the description of the above embodiment, which will not be described in detail here.

[0162] The present invention also discloses a storage medium, which includes a stored program, wherein when the program is run, any one of the above-mentioned methods for automatically positioning educational videos is executed.

[0163] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0164] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0166] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0167] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically positioning educational videos, characterized in that: The following steps are involved: S1. Obtaining uploaded educational resources, wherein the educational resources include educational videos and educational courseware. The educational videos are course videos recorded during teaching, academic lectures, academic conferences, and scientific research training. The educational videos include courseware slideshow content. The educational courseware is courseware that complements the content explained in the educational videos. S2. Extracting video stream features of the educational video based on a deep network model to generate a video key frame sequence, wherein the video key frame sequence includes non-repeated video images; S3, extracting the text sequence in each key frame image based on the optical character recognition method to generate a key frame text sequence; S4, extracting the structured information of the educational courseware, and outputting the content of each page of the courseware in text form to generate a courseware text sequence; S5. Using a courseware-video positioning algorithm on the courseware text sequence and the keyframe text sequence to automatically locate the video explanation position corresponding to each page of the educational courseware, including: S501, obtaining a key frame label list corresponding to a key frame text sequence, wherein the text sequence is frame_kw[i]=[kw1, kw2, ..., kw n ], where i = {1, 2, ..., K}, K represents the number of key frames, n represents the number of texts recognized in the current key frame image, kw n Indicates the nth text recognized by the current key frame image, and the key frame number list corresponding to the key frame text sequence is frame_numlist[i]=[1,2,...,n]; S502. For each key frame image text sequence frame_kw[i], traverse all educational courseware text sequences ppt_kw[j], j = {1, 2, ..., P}, and perform string fuzzy matching on each text. If the string matching similarity is greater than the set threshold, output the key frame label list frame_numlist[i] corresponding to the string. Otherwise, output 0 to obtain the label list of the jth page courseware corresponding to the key frame i. Where ppt_kw[j]=[kw1,kw2,...,kw m ], represents the text sequence of the jth page of courseware, j = {1, 2, ..., P}, P represents the total number of pages of courseware, m represents the number of texts recognized in the courseware on this page, kw m Indicates the mth text recognized by the courseware on this page; S503, list of labels for the jth page of courseware relative to key frame i After processing, when When there are two or more consecutive 0s in the list, only one 0 in the list is retained to obtain a new label list For each frame_numlist[i], traverse all new label lists Calculate the similarity between the two and return the courseware corresponding to the maximum similarity as the courseware page number matched by key frame i; S504, traverse each key frame image, repeat S503, obtain the courseware page numbers that all key frame images match, sort the key frame groups in ascending order by the courseware page numbers, and return the courseware page numbers that all key frame images match; S505: Post-process the courseware page numbers matched to all keyframe images to find the longest increasing subsequence of the keyframe sequence number list corresponding to the sorted courseware page numbers; S506. After sorting by the longest increasing subsequence, only the key frame with the smallest sequence number in each group is retained as the key frame image matching each page of courseware, and the timestamp of the key frame image is used as the video start time for locating the page of courseware.

2. The method for automatically positioning an educational video according to claim 1, characterized in that: S2. Extracting video stream features of the educational video based on a deep network model to generate a video key frame sequence, including: S201, performing frame processing on the educational video to generate frame images with time stamps; S202, extracting image features of the framed image through a convolutional neural network; S203, defining the first framed image as a key frame; S204: Calculate similarity of features of adjacent frame images based on the image features. If the similarity is less than a preset threshold, use the next frame as a key frame, and use the time point of the key frame in the video as the timestamp of the key frame. Otherwise, move to the next image and continue calculating the similarity. S205 , output the key frame image sequence number frame_id and the key frame timestamp frame_ts.

3. The method for automatically positioning an educational video according to claim 2, wherein: The convolutional neural network is one of VGG, GoogleNet, ResNet, DenseNet, MobileNet and ShuffleNet.

4. The method for automatically positioning an educational video according to claim 1, wherein: S3. Extracting text sequences from each keyframe image based on optical character recognition methods, including: S301, pre-processing the key frame image; S302, performing text detection on the pre-processed key frame image, and returning the position coordinates of the line where the text is located in the form of a rectangular frame; S303, performing text recognition based on the text detection, and converting the rectangular frame area into text; S304: Sort all recognized texts in ascending order according to the coordinates of their upper left corners. The text sequence of the i-th key frame image is represented as frame_kw[i]=[kw1, kw2, ..., kw n ], where i = {1, 2, ..., K}, K represents the number of key frames, n represents the number of texts recognized in the current key frame image, kw n Indicates the nth text recognized in the previous keyframe image.

5. The method for automatically positioning an educational video according to claim 4, characterized in that: Perform text detection on the preprocessed keyframe images, including using one of the following methods: Faster R-CNN, FCN, RRPN, TextBoxes, CTPN, and SegLink.

6. The method for automatically positioning an educational video according to claim 1, wherein: S4. Extracting the structured information of the educational courseware and outputting the content of each page of the courseware in text form, including: S401, reading an educational courseware document, calling a document analyzer to parse the educational courseware document, and returning the types and text position coordinates of all objects contained in each page of the courseware, wherein the types of the objects include text, images, tables, and curves, and the text position coordinates are represented by the coordinates of a rectangular box of a text object area; S402, using an optical character recognition method to perform text recognition on non-text objects in the educational courseware, and directly reading the text content of text objects in the educational courseware; S403, sort all recognized texts in ascending order according to the coordinates of their upper left corners, and the text sequence of the j-th page of courseware is expressed as ppt_kw[j]=[kw1, kw2, ..., kw m ], where j = {1, 2, ..., P}, P represents the total number of pages of the courseware, m represents the number of texts recognized in the courseware on this page, kw m Indicates the mth text recognized by the courseware on this page.

7. An automatic positioning device for educational videos, used to implement the automatic positioning method for educational videos according to any one of claims 1 to 6, characterized in that: include: An educational resource upload module, which is used to obtain uploaded educational resources, including educational videos and educational courseware. The educational videos are course videos recorded during teaching, academic lectures, academic conferences, and scientific research training. The educational videos include courseware slideshow content. The educational courseware is courseware that matches the content explained in the educational videos. A video key frame generation module, which is used to extract video stream features of the educational video based on a deep network model to generate a video key frame sequence, wherein the video key frame sequence includes non-repeated video images; A key frame text recognition module is used to extract the text sequence in each key frame image based on the optical character recognition method to generate a key frame text sequence; A courseware structured extraction module is used to extract the structured information of the educational courseware and output the content of each page of the courseware in text form to generate a courseware text sequence; The courseware-video positioning module is used to automatically locate the video explanation position corresponding to each page of the educational courseware by using the courseware-video positioning algorithm on the courseware text sequence and the key frame text sequence.

8. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the method for automatically positioning educational videos described in any one of claims 1 to 6 is executed.

Images