A method for locating knowledge points in course video clips

Abstract

The present invention relates to the technical field of knowledge point positioning, and in particular to a method for positioning knowledge points in course video clips. It includes video frame extraction, in the case of a given course video, video frames are extracted, and by calculating the cosine similarity of consecutive video pixels, pictures with close pixels are merged; the key frame OCR results after merging; the topic extraction of the key frame, using the word segmentation + part of speech method to obtain the word segmentation structure, and then selecting continuous keywords from the abstract to merge, and cleaning stop words and Chinese prefixes and suffixes. The present invention combines the advantages of both pixel similarity and text processing, greatly reducing the number of OCR recognition times, while using the processing ideas of merging, identifying, merging, cleaning, and reconstructing, cleverly avoiding the difficulty of directly locating video clips of knowledge points, and using the text similarity of key frames and knowledge points to well solve the problem of knowledge point video positioning in online education scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge point positioning, and in particular to a method for positioning knowledge points in course video clips. Background Art

[0002] Currently, the technology for locating video clips for knowledge points in online course learning mainly relies on video summary extraction technology. Most of them use pixel differences between each frame of the video or shot segmentation to merge videos. The current general process is as follows: extract frames of video data >> cluster the videos (image similarity or clustering) >> extract video summaries >> compare the similarity between knowledge points and summaries to locate video clips;

[0003] For example, a method for extracting a video summary with application number CN201710035223.5 includes calculating the HSV histogram of all frame images in a video segment to be extracted; calculating the similarity between two adjacent frame images; detecting shot transitions using an adaptive local double-threshold method, cutting the video where shot transitions occur, and finally forming a shot set; using an adaptive threshold-based clustering method to organize similar shots into a cluster set; taking a shot cluster and performing key frame extraction; repeating S6 until key frame extraction has been completed for all shot clusters; and combining the obtained key frames in chronological order to form a final video summary.

[0004] However, excessive reliance on the merging of video pixels may ignore the textual information contained in the video itself. At the same time, when extracting the summary, all the textual information of the merged video clips is used as the summary, ignoring that in actual application scenarios, only the video title is needed without too much redundant information. Summary of the Invention

[0005] This invention is a method for extracting key summaries from video courses in online teaching platforms for student learning. It first extracts frames from the video, merges similar frames, and then performs OCR recognition, significantly reducing the number of OCR recognition attempts. Natural language processing techniques are then used to merge, split, combine, and remove duplicates to extract key frame themes from the video. This ensures accuracy while allowing students to quickly locate the video clips of the key knowledge points being learned during actual course learning.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention provides a method for locating knowledge points in a course video clip, which is characterized by comprising the following steps:

[0008] S100. Video frame extraction: Given a course video, extract frames from the video and merge images with similar pixels by calculating the cosine similarity of consecutive video pixels;

[0009] S200. OCR results of the merged keyframes;

[0010] S300. Extract the keyframe topic by using the word segmentation + part-of-speech method to obtain the word segmentation structure, then select consecutive keywords from the abstract and merge them, and remove stop words and Chinese prefixes and suffixes to obtain the keyframe topic;

[0011] S400. Topic cleaning and reconstruction: Remove duplicate topics from keyframes and use Chinese prefix and suffix syntax and TF-IDF scores to clean and reconstruct them to achieve sentence coherence.

[0012] S500. Knowledge point theme similarity positioning: Use the knowledge point itself and the key frame theme to perform text comparison, sort by similarity score, and select the video clip with the highest similarity as the final positioning of the knowledge point teaching video.

[0013] Furthermore, the video frame extraction is performed in a manner of one frame per second.

[0014] Furthermore, the step of merging pictures with similar pixels by calculating cosine similarity of consecutive video pixels includes:

[0015] S110. Convert each frame of the image into a pixel matrix and tile the matrix to form a vector;

[0016] S120. Calculate the cosine similarity of consecutive video pixels and set the threshold to 0.9 for merging.

[0017] Furthermore, the OCR results of the merged key frames include:

[0018] S210. The image is divided into 9 spaces according to the position by cutting;

[0019] S220. Divide the text information of five areas in the nine spaces into a series of candidate topics, and determine the interval in which the largest proportion of OCR positioning is located;

[0020] S230. Filter the text according to its rotation angle, clarity, and position to obtain important content of the video key frame text, which is used to further merge the video information to obtain key frames and key frame summaries.

[0021] Furthermore, the nine spaces in step S210 are upper left, upper center, upper right, center left, center center, center right, lower left, lower center, and lower right.

[0022] Furthermore, in step S220, the text information is divided into five areas in the nine spaces based on the principle that the closer the text is to the center, the more important the explanation is, and the text information is divided into upper middle, middle middle, upper left, middle left, and middle right.

[0023] Furthermore, in step S110, the pixel values ​​of the image form an n*m matrix, which is converted into a one-dimensional vector with a length of n*m.

[0024] Furthermore, in step S210, the process of dividing the image into 9 spaces according to position by cropping is specifically as follows: the image is divided into nine squares in a manner that the boundary accounts for 0.24 to 0.26, that is, the left, right, top and bottom account for 0.24 to 0.26 respectively, and the middle part accounts for 0.48 to 0.52, divided into upper left, upper middle, upper right, middle left, middle middle, middle right, lower left, lower middle, and lower right.

[0025] Furthermore, in step S210, the process of dividing the image into 9 spaces according to position by cropping is specifically as follows: the image is divided into a nine-square grid in such a way that the boundary accounts for 0.25, that is, the left, right, top and bottom account for 0.25 respectively, and the middle part accounts for 0.5, and is divided into upper left, upper middle, upper right, middle left, middle middle, middle right, lower left, lower middle, and lower right.

[0026] Furthermore, the TF-IDF score is calculated as follows:

[0027] TF-IDF=TF*IDF

[0028] The present invention has at least the following beneficial effects:

[0029] The present invention addresses the defects of current technology that over-rely on key frame merging of video pixels and over-rely on OCR text information to extract summary topics, resulting in inaccurate key frame extraction and unusable summary in actual business. The present invention combines the advantages of pixel similarity and text processing, greatly reducing the number of OCR recognition times. At the same time, it uses the processing ideas of merging, identifying, merging, cleaning and reconstructing to cleverly avoid the difficulty of directly locating video clips of knowledge points, and uses the text similarity of key frame topics and knowledge points to effectively solve the problem of knowledge point video positioning in online education scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 This is the flow chart of the existing patent CN106851437A;

[0032] Figure 2 This is a flow chart of video frame extraction of the present invention;

[0033] Figure 3 A schematic diagram of obtaining video key frame texts according to the present invention;

[0034] Figure 4 Flowchart of the method of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0036] This invention is a method for extracting key summaries from video courses in online teaching platforms for student learning. It first extracts frames from the video, merges similar frames, and then performs OCR recognition, significantly reducing the number of OCR recognition attempts. Natural language processing techniques are then used to merge, split, combine, and remove duplicates to extract key frame themes from the video. This ensures accuracy while allowing students to quickly locate the video clips of the key knowledge points being learned during actual course learning.

[0037] See also Figure 4 As shown, the process of locating the knowledge points in the video clips of the present invention is as follows:

[0038] S100. Video frame extraction: Given a course video, extract frames from the video and merge images with similar pixels by calculating the cosine similarity of consecutive video pixels;

[0039] Among them, the video frame extraction adopts the method of extracting one frame per second;

[0040] In the above steps, by calculating the cosine similarity of consecutive video pixels, images with similar pixels are merged, including:

[0041] First, convert each frame of the image into a pixel matrix, and then tile the matrix to form a vector. The process is as follows:

[0042] The process is as follows: the image is composed of pixels, and the pixel values ​​form an n*m matrix, which is converted into a one-dimensional vector with a length of n*m. For example, the vector is: Tiled into vectors by columns: [1,4,2,5,3,6];

[0043] Then, the cosine similarity of consecutive video pixels is calculated, and the threshold is set to 0.9 for merging. Specifically, the consecutive frame images with similar content, similar pixels, and roughly similar content are merged.

[0044] S200. OCR results of the merged keyframes;

[0045] The specific OCR results of the merged keyframes include:

[0046] The image is divided into 9 spaces according to the position by cutting. The specific process is as follows:

[0047] The image is divided into nine squares with the border accounting for 0.25%, that is, the left, right, top and bottom account for 0.25 respectively, and the middle part accounts for 0.5, which is divided into upper left, upper middle, upper right, middle left, middle middle, middle right, lower left, lower middle, and lower right;

[0048] The text information in five areas of the nine spaces is divided into a series of topic candidates, which is determined by the interval with the largest proportion of OCR positioning positions;

[0049] Filter text by rotation angle, clarity, and position to obtain important content of video keyframe text, which is then used to further merge video information to obtain keyframes and keyframe summaries.

[0050] Among them, the text information in five areas of the nine spaces is divided according to the principle that the closer the text is to the center, the more important the explanation is. The text information is divided into upper middle, middle middle, upper left, middle left, and middle right.

[0051] S300. Extract the keyframe topic by using the word segmentation + part-of-speech method to obtain the word segmentation structure, then select consecutive keywords from the abstract and merge them, and remove stop words and Chinese prefixes and suffixes to obtain the keyframe topic;

[0052] S400. Topic cleaning and reconstruction: Remove duplicate topics from keyframes and use Chinese prefix and suffix syntax and TF-IDF scores to clean and reconstruct them to achieve sentence coherence.

[0053] Among them, the cleaning of Chinese prefixes and suffixes is to clean and reconstruct the commonly used prefixes and suffixes such as *section, *chapter, first*, etc.

[0054] Furthermore, the TF-IDF score is calculated as follows:

[0055] TF-IDF=TF*IDF

[0056] A high score indicates that the word is more important relative to the document, where TF is term frequency and IDF is inverse document frequency;

[0057] For example, if a document contains 100 words and the word "cow" appears 3 times, then the word frequency of "cow" in the document is 3 / 100 = 0.03. One way to calculate the inverse document frequency (IDF) is to divide the total number of documents in the document set by the number of documents in which the word "cow" appears. So, if the word "cow" appears in 1,000 documents and the total number of documents is 10,000,000, its inverse document frequency is lg(10,000,000 / 1,000) = 4. The final TF-IDF score is 0.03*4=0.12;

[0058] S500. Knowledge point theme similarity positioning: Use the knowledge point itself and the key frame theme to perform text comparison, sort by similarity score, and select the video clip with the highest similarity as the final positioning of the knowledge point teaching video.

[0059] Now Figure 3 For reference, an embodiment of the present application is given:

[0060] The knowledge point location process in the video clip of the present invention is roughly divided into four parts: video frame extraction, text similarity merging key frames, key frame theme extraction, theme cleaning and reconstruction, and knowledge point theme similarity location. The specific process is as follows:

[0061] First, see Figure 2 As shown, video frame extraction is performed: given a course video, the video is framed, one frame per second, the image is converted into a pixel matrix, the matrix is ​​tiled to form a vector, the cosine similarity of consecutive video pixels is calculated, and the threshold is set to 0.9 for merging;

[0062] Next, see Figure 3 As shown, the key frames are merged by text similarity: according to the OCR results of the key frames after the above merger, the image is divided into 9 spaces according to the position, namely the upper left, upper middle, upper right, middle left, middle center, middle right, lower left, lower middle, and lower right nine areas. According to the rules of teaching videos, the closer the text is to the center, the more important the explanation is. The text information of the upper middle, middle center, upper left, middle left, and middle right is divided into themes to be selected in a tiered manner, and the interval with the largest proportion of OCR positioning is determined; at the same time, the text is filtered according to the rotation angle, clarity, and position to obtain the important content of the video key frame text, which is used to further merge the video information to obtain key frames and key frame summaries, such as Figure 3 As shown, Figure 3 The texts that meet the conditions in the 9 areas are as follows: [Ethernet frame attributes - Ethernet frame size] - upper medium, [The Ethernet standard and IEEE802.3 standard define the smallest frame as] - medium medium;

[0063] Next, we extract the keyframe topics. After obtaining the video keyframes through image and text similarity, we extract the topics of each keyframe. The main idea is to use word segmentation and part-of-speech to obtain the word segmentation structure. Then, we select consecutive keywords from the abstract and merge them. We also remove stop words and Chinese prefixes and suffixes to obtain the keyframe topics.

[0064] Next, we clean and reconstruct the topics. The topics of the keyframes mentioned above may still be repeated or unavailable. Therefore, in this stage, we remove duplicate keyframes based on themes, obtaining video keyframes with larger intervals and fewer segments. We also use Chinese prefix and suffix syntax and TF-IDF scores to clean and reconstruct the keyframes to achieve sentence coherence.

[0065] Finally, knowledge point theme similarity positioning: Finally, use the knowledge point itself and the key frame theme to perform text comparison, sort by similarity score, and select the video clip with the highest similarity as the final positioning of the knowledge point teaching video.

[0066] This application addresses the defects of current technology that over-rely on key frame merging of video pixels and over-rely on OCR text information to extract summary topics, resulting in inaccurate key frame extraction and unusable summary in actual business. The present invention combines the advantages of pixel similarity and text processing, greatly reducing the number of OCR recognition times. At the same time, it uses the processing ideas of merging, identifying, merging, cleaning, and reconstructing to cleverly avoid the difficulty of directly locating video clips of knowledge points, and uses the text similarity of key frame topics and knowledge points to effectively solve the problem of knowledge point video positioning in online education scenarios.

[0067] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for locating knowledge points in a course video clip, characterized in that: The following steps are involved: S100. Video frame extraction: Given a course video, extract frames from the video and merge images with similar pixels by calculating the cosine similarity of consecutive video pixels; S200. OCR results of the merged keyframes; S300. Extract the keyframe topic by using the word segmentation + part-of-speech method to obtain the word segmentation structure, then select consecutive keywords from the abstract and merge them, and remove stop words and Chinese prefixes and suffixes to obtain the keyframe topic; S400. Topic cleaning and reconstruction: Remove duplicate topics from keyframes and use Chinese prefix and suffix syntax and TF-IDF scores to clean and reconstruct them to achieve sentence coherence. S500. Knowledge point theme similarity positioning: Use the knowledge point itself and the key frame theme to perform text comparison, sort by similarity score, and select the video clip with the highest similarity as the final positioning of the knowledge point teaching video; The video frame extraction is performed in a manner of one frame per second; The method of merging pictures with similar pixels by calculating the cosine similarity of consecutive video pixels includes: S110. Convert each frame of the image into a pixel matrix and tile the matrix to form a vector; S120. Calculate the cosine similarity of consecutive video pixels and set the threshold to 0.9 for merging; The OCR results of the merged key frames include: S210. Divide the image into 9 spaces according to position by cutting; S220. Divide the text information into five areas in the nine spaces and fill in the topic candidates in a tiered manner, according to which interval the largest proportion of OCR positioning positions is determined, wherein the five areas are: upper center, middle center, upper left, middle left, and middle right; S230. Filter the text according to its rotation angle, clarity, and position to obtain important content of the video key frame text, which is used to further merge the video information to obtain key frames and key frame summaries.

2. The method for locating knowledge points in course video clips according to claim 1, characterized in that: The nine spaces in step S210 are upper left, upper center, upper right, center left, center center, center right, lower left, lower center, and lower right.

3. The method for locating knowledge points in course video clips according to claim 2, characterized in that: In step S220, the text information of the five areas in the nine spaces is divided into upper middle, middle middle, upper left, middle left, and middle right according to the principle that the text closer to the center is the focus of the explanation.

4. The method for locating knowledge points in course video clips according to claim 1, characterized in that: In step S110 , the pixel values ​​of the image form an n*m matrix, which is converted into a one-dimensional vector with a length of n*m.

5. The method for locating knowledge points in course video clips according to claim 2, characterized in that: In step S210, the process of dividing the image into 9 spaces according to position by cropping is specifically as follows: the image is divided into nine squares in a manner that the boundary accounts for 0.24~0.26, that is, the left, right, top and bottom account for 0.24~0.26 respectively, and the middle part accounts for 0.48~0.52, divided into upper left, upper middle, upper right, middle left, middle middle, middle right, lower left, lower middle, and lower right.

6. The method for locating knowledge points in course video clips according to claim 5, characterized in that: In step S210, the process of dividing the image into 9 spaces according to position by cutting is specifically as follows: the image is divided into nine squares in a manner that the boundary accounts for 0.25, that is, the left, right, top and bottom account for 0.25 respectively, and the middle part accounts for 0.5, divided into upper left, upper middle, upper right, middle left, middle middle, middle right, lower left, lower middle, and lower right.

7. The method for locating knowledge points in course video clips according to claim 1, characterized in that: The TF-IDF score is calculated as follows: TF-IDF = TF * IDF.

Images