Academic video key frame extraction method, device and equipment based on video engine

Abstract

The application discloses a kind of based on academic video key frame extraction method, device and equipment of video engine, it is related to video image processing technical field.Method includes obtaining the video meta information of academic video;According to frame extraction time interval, academic video is sequentially framed to generate picture sequence numbered according to time sequence;First picture is extracted from picture sequence, and the comprehensive information richness of first picture is determined based on the text information density and visual element richness of first picture;According to the comprehensive information richness, determine initial jump step;Second picture is determined based on the picture number of first picture and initial jump step;Similarity comparison is carried out to first picture and second picture;If similarity result is not greater than preset similarity threshold, then second picture is used as key frame and is stored in key frame set.Through the comprehensive information richness of first picture, jump step is determined, the comparison amount of redundant frame is reduced, and the extraction efficiency of key frame is improved.

Description

Technical Field

[0001] This invention relates to the field of video image processing technology, and specifically to a method, apparatus, and device for extracting keyframes from academic videos based on a video engine. Background Technology

[0002] The video engine is a modular, schedulable, underlying software core framework that can fully cover the entire lifecycle of video processing. It is responsible for the unified scheduling of the entire process, including video reading, decoding, frame extraction, image preprocessing, multi-threaded scheduling, rendering / exporting, and task status management. It provides standardized calling interfaces to the outside world and is the underlying driving core of video processing business.

[0003] Academic videos refer to video recordings made primarily for academic teaching, project presentations, industry discussions, and research dissemination. The visual medium for academic videos is typically PowerPoint presentations (PPT), and during playback, there are instances where single slides remain static for extended periods, content changes abruptly when turning pages, and there is a high density of text, planning diagrams, and charts.

[0004] Because the original academic videos contain a large number of video frames and require a lot of storage, in order to fully restore the academic information contained in the original academic videos with a small number of static images, keyframes are usually extracted from the original academic videos and merged into a document carrier (such as PDF, Portable Document Format) for circulation and archiving.

[0005] Currently, keyframe extraction based on video engines typically employs frame-by-frame similarity comparison detection logic. This approach requires performing complete feature comparison calculations on every pair of extracted frames. Since academic video slides generate a large number of highly similar redundant frames during static periods, frame-by-frame comparison would repeatedly perform similarity calculations on numerous repetitive images, resulting in unnecessary computational overhead and increasing overall processing time. Summary of the Invention

[0006] The purpose of this invention is to provide a method, apparatus, and device for extracting keyframes from academic videos based on a video engine. By determining the jump step size through the comprehensive information richness of the first image, the amount of redundant frames compared is reduced, thus solving the problem of low efficiency in existing frame-by-frame comparison extraction of keyframes.

[0007] This invention is achieved through the following technical solution:

[0008] The first aspect of this application provides a method for extracting keyframes from academic videos based on a video engine, including:

[0009] Obtain video metadata of the academic video, and determine the frame extraction time interval of the academic video based on the metadata;

[0010] The academic video is sequentially frame-sampling according to the frame-sampling time interval to generate a sequence of images numbered in chronological order.

[0011] Extract the first image from the image sequence, and determine the comprehensive information richness of the first image based on its text information density and visual element richness.

[0012] The initial jump step size is determined based on the comprehensive information richness and the preset mapping relationship between comprehensive information richness and jump step size.

[0013] The second image is determined based on the image number of the first image and the initial jump step size;

[0014] The first image and the second image are compared for similarity to obtain a similarity result. If the similarity result is not greater than the preset similarity threshold, the second image is used as a keyframe and stored in the keyframe set.

[0015] In one feasible implementation, the method further includes:

[0016] After determining the second image as a keyframe, the second image is used as the new first image. Based on the number of the new first image and the initial jump step size of the corresponding new first image, the new second image is determined, and the similarity between the new first image and the new second image is compared.

[0017] In one feasible implementation, the method further includes:

[0018] After determining that the second image is a keyframe, the value of the initial jump step size is reduced to determine a new jump step size;

[0019] Based on the number of the first image and the new jump step size, determine the new second image;

[0020] Compare the similarity between the first image and the new second image.

[0021] In one feasible implementation, the method further includes:

[0022] If the similarity result is greater than the preset similarity threshold, the second image is used as the new first image. The new second image is determined based on the image number of the new first image and the initial jump step size of the corresponding new first image. The similarity between the new first image and the new second image is compared to extract keyframes.

[0023] In one feasible implementation, the similarity comparison between the first image and the second image includes:

[0024] Extract the number of characters from the first and second images;

[0025] If the number of characters exceeds a preset threshold, the similarity between the first image and the second image is compared based on a similarity algorithm of the text content.

[0026] If the number of characters does not exceed a preset character count threshold, then a similarity algorithm based on image features is used to compare the similarity between the first image and the second image.

[0027] In one feasible implementation, the method further includes:

[0028] The extracted keyframes are compared with images in a preset exception image library for similarity.

[0029] If the similarity comparison result exceeds the preset exception exclusion threshold, the keyframe will be removed from the keyframe set.

[0030] In one feasible implementation, the method further includes:

[0031] The image sequence is divided into multiple consecutive sub-image sequences;

[0032] Extract the first image from each sub-image sequence, perform a similarity comparison between the first image and the second image, and merge the keyframes extracted from each sub-image sequence into a keyframe set in chronological order.

[0033] In one feasible implementation, the method further includes:

[0034] Based on the characteristics of academic videos, local detection windows are set for the frame space variation parts. The window region images of the first image and the second image are extracted, and the similarity between the local detection windows of the first image and the local detection windows of the second image is compared.

[0035] A second aspect of this application provides a device for extracting keyframes from academic videos based on a video engine, comprising:

[0036] A frame extraction interval determination unit is used to acquire video metadata of the academic video and determine the frame extraction time interval of the academic video based on the metadata.

[0037] The image sequence generation unit is used to sequentially extract frames from the academic video according to the frame extraction time interval to generate an image sequence numbered in chronological order.

[0038] The comprehensive information richness determination unit is used to extract a first image from the image sequence and determine the comprehensive information richness of the first image based on the text information density and visual element richness of the first image.

[0039] The initial jump step size determination unit is used to determine the initial jump step size based on the comprehensive information richness and the preset mapping relationship between comprehensive information richness and jump step size.

[0040] The second image determination unit determines the second image based on the image number of the first image and the initial jump step size;

[0041] The keyframe extraction unit is used to compare the similarity between the first image and the second image to obtain a similarity result; if the similarity result is not greater than a preset similarity threshold, the second image is used as a keyframe and stored in the keyframe set.

[0042] A third aspect of this application provides an electronic device, including: a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the above-described method.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] This application embodiment uses a video engine to parse video metadata to extract frames and obtain an ordered image sequence. The comprehensive information richness is calculated by analyzing the text information density and visual element richness of the first image (base frame). Then, based on the mapping relationship between comprehensive information richness and jump step size, an initial jump step size is adaptively matched. A second image (interval frame) is selected according to this jump step size interval. The similarity between the base frame and the interval frame is compared, and frames with differences exceeding a threshold are included as keyframes. Since the comprehensive information richness of an image is directly proportional to the narration duration, a larger initial jump step size is matched to images with high comprehensive information richness, reducing the number of comparisons of redundant frames and improving the efficiency of keyframe extraction. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0046] Figure 1 A flowchart illustrating the method for extracting keyframes from academic videos based on a video engine, as provided in this application embodiment;

[0047] Figure 2 A flowchart illustrating a specific implementation of the extraction method provided in this application embodiment;

[0048] Figure 3 This is a schematic diagram of the similarity comparison process in the extraction method provided in the embodiments of this application;

[0049] Figure 4 A schematic diagram of the structure of the academic video keyframe extraction device based on a video engine provided in the embodiments of this application;

[0050] Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0052] As will be known to those skilled in the art, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0053] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such process, method, product, or apparatus.

[0054] Example 1:

[0055] Embodiment 1 of this application provides a method for extracting keyframes from academic videos based on a video engine, in order to solve the problem of low efficiency in existing frame-by-frame comparison keyframe extraction.

[0056] The subject executing this method can be any computing device capable of implementing the method, such as a server, mobile phone, personal computer, smart wearable device, smart robot, etc.

[0057] Furthermore, the embodiments of this application do not limit the execution order of different steps. When using the method provided in the embodiments of this application, the execution order of different steps can be adjusted according to actual needs.

[0058] For ease of description, the following uses a video engine-based academic video keyframe extraction device as the execution subject of this method to provide a detailed description of the method provided in this application embodiment.

[0059] like Figure 1The diagram shown is a flowchart illustrating the specific implementation of a method for extracting keyframes from academic videos based on a video engine, as provided in this application embodiment. The method includes the following steps 11-16:

[0060] Step 11: Obtain video metadata of the academic video, and determine the frame extraction time interval of the academic video based on the metadata.

[0061] Video metadata includes information such as video encoding format, resolution, frame rate, total number of frames, duration, and image characteristics.

[0062] This embodiment uses a video engine to parse academic videos to obtain video metadata. Specifically, it can be:

[0063] After the video engine loads the target academic video file, the engine's internal decapsulation module reads the video encapsulation layer data and extracts basic metadata, including video encoding format, resolution, frame rate, total number of frames, and total duration. Then, the engine's analysis module statistically analyzes image features such as screen change frequency, text density, and color distribution, and integrates all the data to form complete video metadata, which is used for subsequent adaptive setting of frame extraction time intervals.

[0064] Subsequent processing parameters are set based on video clarity, image features, etc., such as frame extraction time interval T, detection window size and position, and similarity threshold θ.

[0065] The frame extraction time interval refers to the basic time length between two adjacent frame extraction operations when the video engine performs sampling and frame extraction on an academic video.

[0066] The detection window is a pre-defined local image region on the extracted frames of the image, used to limit the range of feature calculation when comparing the similarity of two images. During similarity calculation, only image features within this window are extracted and compared; image regions outside the window are not included in the similarity calculation.

[0067] Step 12: Sequentially extract frames from the academic video according to the frame extraction time interval to generate a sequence of images numbered in chronological order.

[0068] Following the frame extraction time interval determined in step 11, the academic video is sequentially extracted along the timeline to generate an image sequence F={f1, f2, f3, ..., f n}, where n is the total number of images extracted, and each image is numbered in chronological order.

[0069] Step 13: Extract the first image from the image sequence, and determine the comprehensive information richness of the first image based on the text information density and visual element richness of the first image.

[0070] The first image, also known as the reference frame, is typically the first frame (f1) in the image sequence when it is first extracted. The extracted first image is then stored in the keyframe set.

[0071] Text information density represents the proportion of effective text content within a single image, reflecting the amount of text information in a slideshow.

[0072] The determination of text information density can be achieved by performing text detection and OCR (Optical Character Recognition) on the first image, counting the total pixel area of ​​the effective text region in the image, and dividing the total pixel area of ​​the text region by the total number of pixels in the entire image to obtain the text information density; or by counting the total number of recognized effective text characters, and performing normalization calculations based on the image resolution to obtain the text information density value in the 0-1 range.

[0073] Visual element richness characterizes the amount of non-textual visual content in a single image, and is used to measure the complexity of non-textual visual elements such as drawings, charts, diagrams, and graphic color blocks.

[0074] The determination of visual element richness can be as follows: perform image feature detection on the first image to identify non-textual visual targets such as graphics, charts, lines, and color blocks on paper, count the pixel area occupied by the above visual elements, and divide it by the total number of pixels in the image to obtain the proportion; or it can be as follows: through the statistics of image edge features and texture features, after normalization processing, output the visual element richness value in the range of 0-1.

[0075] The overall information richness characterizes the fullness of the effective academic information in the current image.

[0076] In this embodiment, the comprehensive information richness is determined by weighted fusion of text information density and visual element richness. Specifically, it can be expressed as:

[0077] ;

[0078] In the formula, , These represent text information density and visual element richness, respectively. , The preset weighting coefficients are used, and they satisfy the following conditions: + It can be pre-configured according to the type of academic video; for planning and reporting academic videos, the weight β corresponding to the richness of visual elements can be appropriately increased, with emphasis on taking into account the information of drawings and charts.

[0079] Step 14: Determine the initial jump step size based on the comprehensive information richness and the preset mapping relationship between comprehensive information richness and jump step size.

[0080] Generally, the higher the overall information richness, the longer the time spent explaining the current image, and the longer the image is displayed in the academic video. Therefore, in the pre-set mapping relationship between overall information richness and jump step size, overall information richness and jump step size are positively correlated, that is, the higher the overall information richness, the larger the corresponding jump step size.

[0081] The mapping relationship can be in the form of a function expression of comprehensive information richness and jump step size, or it can be in the form of a queryable table, etc. The corresponding jump step size can be determined based on the comprehensive information richness.

[0082] In this step, based on the comprehensive information richness determined in step 13 and the mapping relationship, the initial jump step size for the corresponding first image is determined.

[0083] It should be noted that the relationship between comprehensive information richness and jump step size can be one-to-one or one-to-many. That is, for a given comprehensive information richness, there can be one jump step size or multiple jump step sizes.

[0084] The selection of the jump step size can be based on the specific application scenario, such as the keyframe extraction mode and / or the nature of the academic video. For example, under normal circumstances, the normal mode is used: based on a one-to-one relationship, the jump step size corresponding to the information richness of the current image is determined; if a keyframe set needs to be obtained quickly, the fast mode is used: based on a one-to-many relationship, the larger jump step size is selected from multiple jump step sizes; if the academic video is an important academic presentation, and every keyframe in the academic video needs to be extracted (without omission), the precise mode is used: based on a one-to-many relationship, the larger jump step size is selected from multiple jump step sizes.

[0085] Therefore, the selection of the jump step size can be determined comprehensively based on the information richness of the current image and the application scenario.

[0086] Step 15: Determine the second image based on the image number of the first image and the initial jump step size.

[0087] The sum of the first image number and the initial jump step size is used as the number of the second image, thus determining the second image. For example, if the first image number is 1 (i.e., the first image is f1) and the initial jump step size is 8, then the second image number is 1 + 8 = 9, meaning the second image is f1. .

[0088] Step 16: Compare the similarity between the first image and the second image to obtain the similarity result; if the similarity result is not greater than the preset similarity threshold, then the second image is used as a keyframe and stored in the keyframe set.

[0089] If the similarity result of the first image and the second image is not greater than the preset similarity threshold, it means that the first image and the second image are not similar. Therefore, the second image is stored as a keyframe in the keyframe set.

[0090] In one feasible implementation, after determining the second image as a keyframe, if in precise mode, a backtracking comparison is required to avoid missing keyframes between the first and second images. The specific operation could be:

[0091] Decrease the initial jump step size to determine a new jump step size; determine a new second image based on the first image number and the new jump step size; compare the similarity between the first image and the new second image.

[0092] The new jump step size can be half of the initial jump step size. For example, if the initial jump step size is 8, the new jump step size is 4, and the new second image is... ,Compare and The similarity, if and If the similarity is still not greater than the preset similarity threshold, then... Store in the keyframe set and continue comparing. and The similarity between the first image and its adjacent images is compared until the similarity between the first image and the adjacent images is reached, at which point the current sub-process ends. If, during the comparison process, the similarity between two images exceeds a preset threshold, it indicates that the images between the two images are redundant frames, and the current sub-process ends directly without further similarity comparison.

[0093] This implementation also includes, after determining the new jump step size to be 4, setting the second image (still using...) (For example) As the new first image, the difference between the new first image and the new jump step size, plus 1, is used as the number of the new second image, that is, the new second image is... ,Compare and If the similarity is still not greater than the preset similarity threshold, then... Store in the keyframe set; until the first image is compared ( The image adjacent to the first image ( If the similarity between two images exceeds a preset threshold, it indicates that the images between them are redundant frames, and the current sub-process ends without further similarity comparison.

[0094] In summary, when the similarity between the first image and the second image is no greater than the preset similarity threshold, the initial jump step size is reduced, and the image between the first image and the second image is compared with the first image and the second image respectively to extract keyframes, so as not to miss the keyframes between the first image and the second image.

[0095] In another feasible implementation, after determining the second image as a keyframe, if in fast mode, the comparison between the first and second images is skipped directly, the second image is used as the new first image, the initial jump step size corresponding to the new first image is determined, and the above steps 14-16 are repeated until all images in the image sequence are compared.

[0096] Using the fast mode can significantly improve the efficiency of keyframe determination, thereby quickly generating PDFs. In this mode, since there is no frame-by-frame comparison, there is a possibility of missing individual slides in academic videos. However, the explanation of the missed slide must have taken up very little time, indicating that it does not contain much useful information. Therefore, in fast mode, important keyframes can still be extracted.

[0097] The above describes the operation process after determining the second image as a keyframe.

[0098] If the similarity between the first image and the second image is greater than the preset similarity threshold, it means that the first image and the second image are similar, and we consider them to be the same image. That is, the slide of this page was not turned in the academic video. Since the first image has already been stored in the keyframe set, the second image is not stored in the keyframe set, and the images between the first image and the second image are no longer compared for similarity. Next, the second image is used as the new first image, and the initial jump step size corresponding to the new first image is determined (since the new first image is the same as the original first image, the newly determined initial jump step size should be the same as the first image; or the jump step size in the previous round of comparison can be directly used). The above steps 14-16 are repeated until all images in the image sequence have been compared.

[0099] Based on a dynamic jump step size, a dynamic sliding window jump comparison method is adopted, which reduces the number of similarity comparisons and improves the extraction efficiency of keyframes compared to the existing frame-by-frame comparison method. Experimental data shows that on five urban planning academic video test sets, the processing time of this embodiment is reduced by an average of about 73.6% compared to the frame-by-frame comparison method.

[0100] like Figure 2 As shown, a specific implementation logic diagram of the above video keyframe extraction method is provided.

[0101] Parameter initialization: i represents the image number, and n represents the total number of images; i starts from 1 and takes values ​​based on the number of images. Determine the initial jump step size steps, calculate the second image number j = i + steps, and extract the image when j is not greater than n. ,Compare and similarity If similarity If the similarity is greater than the preset similarity threshold T, then set i=j and return to the step to determine the jump step size; if the similarity is not greater than the preset similarity threshold, then... Store in a keyframe set, and divide into three sub-processes: Let steps=[ ], return to the steps for calculating the second image number j; let i = j, j = j - [ +1, return to extract The steps are as follows: Let i=j, return to determine the jump step size, end the process when i or j is greater than n, and output the keyframe set.

[0102] In one feasible implementation, the similarity comparison algorithm used in step 16 can be a text-based similarity algorithm or an image-feature-based similarity algorithm.

[0103] like Figure 3 As shown, it specifically includes:

[0104] Extract the number of characters from the first image and the second image; if the number of characters exceeds a preset threshold, use a similarity algorithm based on text content to compare the similarity between the first image and the second image; if the number of characters does not exceed the preset threshold, use a similarity algorithm based on image features to compare the similarity between the first image and the second image.

[0105] The text extraction from the first and second images can be achieved by recognizing the text in the images using OCR (Optical Character Recognition), and counting the number of recognized characters as the text count.

[0106] Similarity algorithms based on text content can be one of edit distance or cosine similarity, while similarity algorithms based on image features can be one or more combinations of color histogram similarity, structural similarity index (SSIM), and perceptual hash similarity.

[0107] In addition, this embodiment also pre-builds an exception image library, which stores user-specified background images and invalid information images (including but not limited to solid color transition frames, intro and outro logo frames, and advertising frames unrelated to video content), and stores them in the except directory as the exception image library.

[0108] For each keyframe extracted in steps 11-16 above, a similarity comparison is performed between each keyframe and an image in a preset exception library; if the similarity comparison result exceeds a preset exception exclusion threshold, the keyframe is removed from the keyframe set.

[0109] For the keyframe set after removing the exception images, merge the keyframes in the set into a PDF document in chronological order and output it to the specified output directory.

[0110] This implementation method can quickly extract lengthy academic videos on urban planning into highly informative PDF texts. The number of keyframes extracted is approximately 9.4% of the original number of frames (compression ratio of approximately 10.6:1), and the information coverage in the keyframe set reaches over 94%, which can meet the needs of academic materials for browsing, annotation, archiving, and sharing.

[0111] In one feasible implementation, to further improve the efficiency of keyframe extraction, this embodiment also includes multi-threaded parallel processing of the similarity comparison process. Specifically, this includes:

[0112] The image sequence is divided into multiple consecutive sub-image sequences; a first image is extracted from each sub-image sequence, and a similarity comparison is performed between the first image and a second image; the keyframes extracted from each sub-image sequence are merged into a keyframe set in chronological order.

[0113] In one feasible implementation, considering the characteristic of urban planning academic videos where the layout of the images is relatively fixed but some content (such as maps, planning drawings, and text titles) changes, this embodiment further includes the following to improve the efficiency and accuracy of similarity comparison:

[0114] Based on the characteristics of academic videos, local detection windows are set for the frame space variation areas. The window regions of the first and second images are captured, and the similarity between the local detection windows of the first and second images is compared. Similarity calculation is limited to the local area covered by the window, ignoring the image area outside the window to eliminate interference from fixed elements such as the background, thus improving the targeting and accuracy of keyframe detection.

[0115] In one feasible implementation, to adapt to long-duration video processing and avoid repetitive calculations after processing interruptions, this embodiment also includes a breakpoint continuation mechanism, specifically including:

[0116] During the processing in steps 13-16, the values ​​of the currently processed frame number i, the jump step size steps, and the list of frame numbers extracted into the keyframe set are recorded in real time. The real-time recorded information is persistently stored in a local progress file. If the processing process is interrupted due to an exception and restarted, the progress file is read first, and processing continues from the recorded position i based on the progress record in the progress file to avoid repeating the processing of completed parts.

[0117] Example 2:

[0118] To address the problem of low efficiency in existing frame-by-frame keyframe extraction methods, and based on the same inventive concept as Embodiment 1, this application also provides a device for extracting keyframes from academic videos based on a video engine.

[0119] A schematic diagram of the specific structure of the device is shown below. Figure 4 As shown, it includes the following functional units:

[0120] The frame extraction interval determination unit 41 is used to acquire video metadata of the academic video and determine the frame extraction time interval of the academic video based on the metadata.

[0121] Image sequence generation unit 42 is used to sequentially extract frames from the academic video according to the frame extraction time interval to generate an image sequence numbered in chronological order.

[0122] The comprehensive information richness determination unit 43 is used to extract a first image from the image sequence and determine the comprehensive information richness of the first image based on the text information density and visual element richness of the first image.

[0123] The initial jump step size determination unit 44 is used to determine the initial jump step size based on the comprehensive information richness and the preset mapping relationship between comprehensive information richness and jump step size.

[0124] The second image determination unit 45 determines the second image based on the image number of the first image and the initial jump step size;

[0125] The keyframe extraction unit 46 is used to compare the similarity between the first image and the second image to obtain a similarity result; if the similarity result is not greater than a preset similarity threshold, the second image is used as a keyframe and stored in the keyframe set.

[0126] The keyframe extraction unit is also used for:

[0127] After determining the second image as a keyframe, the second image is used as the new first image. Based on the number of the new first image and the initial jump step size of the corresponding new first image, the new second image is determined, and the similarity between the new first image and the new second image is compared.

[0128] The keyframe extraction unit is also used for:

[0129] After determining the second image as a keyframe, the value of the initial jump step size is reduced to determine a new jump step size; based on the number of the first image and the new jump step size, a new second image is determined; the similarity between the first image and the new second image is compared.

[0130] The keyframe extraction unit is also used for:

[0131] If the similarity result is greater than the preset similarity threshold, the second image is used as the new first image. The new second image is determined based on the image number of the new first image and the initial jump step size of the corresponding new first image. The similarity between the new first image and the new second image is compared to extract keyframes.

[0132] The keyframe extraction unit is also used for:

[0133] Extract the number of characters from the first image and the second image; if the number of characters exceeds a preset threshold, compare the similarity between the first image and the second image based on a similarity algorithm of text content; if the number of characters does not exceed the preset threshold, compare the similarity between the first image and the second image using an image feature-based similarity algorithm.

[0134] The extraction device in this embodiment further includes a screening unit, specifically used for:

[0135] The extracted keyframes are compared with images in a preset exception library for similarity; if the similarity comparison result exceeds a preset exception exclusion threshold, the keyframes are removed from the keyframe set.

[0136] The extraction device in this embodiment also includes a multi-threaded processing control unit, specifically used for:

[0137] The image sequence is divided into multiple consecutive sub-image sequences; a first image is extracted from each sub-image sequence, and a similarity comparison is performed between the first image and a second image; the keyframes extracted from each sub-image sequence are merged into a keyframe set in chronological order.

[0138] The keyframe extraction unit in this embodiment is also used for:

[0139] Based on the characteristics of academic videos, local detection windows are set for the frame space variation parts. The window region images of the first image and the second image are extracted, and the similarity between the local detection windows of the first image and the local detection windows of the second image is compared.

[0140] This application embodiment uses a video engine to parse video metadata to extract frames and obtain an ordered image sequence. The comprehensive information richness is calculated by analyzing the text information density and visual element richness of the first image (base frame). Then, based on the mapping relationship between comprehensive information richness and jump step size, an initial jump step size is adaptively matched. A second image (interval frame) is selected according to this jump step size interval. The similarity between the base frame and the interval frame is compared, and frames with differences exceeding a threshold are included as keyframes. Since the comprehensive information richness of an image is directly proportional to the narration duration, a larger initial jump step size is matched to images with high comprehensive information richness, reducing the number of comparisons of redundant frames and improving the efficiency of keyframe extraction.

[0141] Based on the same inventive concept as the foregoing embodiments of this application, this application also provides a computing device.

[0142] like Figure 5 As shown, the computing device includes a memory 51 and a processor 52. The memory 51 can be configured to store various other data to support operation on the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device. The memory 51 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0143] The processor 52, coupled to the memory 51, is used to execute the computer program stored in the memory 51 for performing the method for extracting key frames of academic videos based on a video engine as described in the foregoing embodiments.

[0144] When the processor 52 executes the computer program to perform a method for extracting keyframes from academic videos based on a video engine, it parses video metadata through the video engine to extract an ordered sequence of images. It calculates the comprehensive information richness of the first image (reference frame) based on its text information density and visual element richness. Then, it adaptively matches an initial jump step size based on the mapping relationship between the comprehensive information richness and the jump step size. Based on this jump step size interval, it selects a second image (interval frame) and compares the similarity between the reference frame and the interval frame. Images with differences exceeding a threshold are included as keyframes. Since the comprehensive information richness of an image is directly proportional to the presentation duration, matching a larger initial jump step size to images with high comprehensive information richness reduces the number of comparisons of redundant frames and improves the efficiency of keyframe extraction.

[0145] When the processor 52 executes the computer program in the memory 51, in addition to the functions described above, it can also perform other functions, as detailed in the descriptions of the preceding embodiments.

[0146] Furthermore, such as Figure 5 As shown, the computing device also includes other components such as a display 54, a communication component 53, a power supply component 55, and an audio component 56. Figure 5 The diagram only shows some components and does not mean that the computing device includes only these components. Figure 5 The components shown.

[0147] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the methods provided in the above embodiments.

[0148] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0150] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for extracting keyframes from academic videos based on a video engine, characterized in that, include: Obtain video metadata of the academic video, and determine the frame extraction time interval of the academic video based on the metadata; The academic video is sequentially frame-sampling according to the frame-sampling time interval to generate a sequence of images numbered in chronological order. Extract the first image from the image sequence, and determine the comprehensive information richness of the first image based on its text information density and visual element richness. The initial jump step size is determined based on the comprehensive information richness and the preset mapping relationship between comprehensive information richness and jump step size. The second image is determined based on the image number of the first image and the initial jump step size; The similarity between the first image and the second image is compared to obtain the similarity results; If the similarity result is not greater than the preset similarity threshold, the second image will be used as a keyframe and stored in the keyframe set.

2. The method according to claim 1, characterized in that, The method further includes: After determining the second image as a keyframe, the second image is used as the new first image. Based on the number of the new first image and the initial jump step size of the corresponding new first image, the new second image is determined, and the similarity between the new first image and the new second image is compared.

3. The method according to claim 1, characterized in that, The method further includes: After determining that the second image is a keyframe, the value of the initial jump step size is reduced to determine a new jump step size; Based on the number of the first image and the new jump step size, determine the new second image; Compare the similarity between the first image and the new second image.

4. The method according to claim 1, characterized in that, The method further includes: If the similarity result is greater than the preset similarity threshold, the second image is used as the new first image. The new second image is determined based on the image number of the new first image and the initial jump step size of the corresponding new first image. The similarity between the new first image and the new second image is compared to extract keyframes.

5. The method according to claim 1, characterized in that, The step of comparing the similarity between the first image and the second image includes: Extract the number of characters from the first and second images; If the number of characters exceeds a preset threshold, the similarity between the first image and the second image is compared based on a similarity algorithm of the text content. If the number of characters does not exceed a preset character count threshold, then a similarity algorithm based on image features is used to compare the similarity between the first image and the second image.

6. The method according to claim 1, characterized in that, The method further includes: The extracted keyframes are compared with images in a preset exception image library for similarity. If the similarity comparison result exceeds the preset exception exclusion threshold, the keyframe will be removed from the keyframe set.

7. The method according to claim 1, characterized in that, The method further includes: The image sequence is divided into multiple consecutive sub-image sequences; Extract the first image from each sub-image sequence, perform a similarity comparison between the first image and the second image, and merge the keyframes extracted from each sub-image sequence into a keyframe set in chronological order.

8. The method according to claim 1, characterized in that, The method further includes: Based on the characteristics of academic videos, local detection windows are set for the frame space variation parts. The window region images of the first image and the second image are extracted, and the similarity between the local detection windows of the first image and the local detection windows of the second image is compared.

9. A device for extracting keyframes from academic videos based on a video engine, characterized in that, include: A frame extraction interval determination unit is used to acquire video metadata of the academic video and determine the frame extraction time interval of the academic video based on the metadata. The image sequence generation unit is used to sequentially extract frames from the academic video according to the frame extraction time interval to generate an image sequence numbered in chronological order. The comprehensive information richness determination unit is used to extract a first image from the image sequence and determine the comprehensive information richness of the first image based on the text information density and visual element richness of the first image. The initial jump step size determination unit is used to determine the initial jump step size based on the comprehensive information richness and the preset mapping relationship between comprehensive information richness and jump step size. The second image determination unit determines the second image based on the image number of the first image and the initial jump step size; The keyframe extraction unit is used to compare the similarity between the first image and the second image to obtain the similarity result; If the similarity result is not greater than the preset similarity threshold, the second image will be used as a keyframe and stored in the keyframe set.

10. An electronic device, characterized in that, include: A processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method as claimed in any one of claims 1-8.

Images