Video frame accurate positioning method based on hierarchical knowledge base

Abstract

The invention discloses a video frame precise positioning method based on a hierarchical knowledge base, and the method comprises the steps: carrying out the preprocessing of each teaching video of a MOOC class in an online education platform, and extracting a representative key frame and a precise timestamp of the representative key frame in an original video; on this basis, knowledge points are extracted by means of a visual language large model, and a hierarchical knowledge base structure is constructed; then, carrying out deep semantic analysis on a natural language question input by a user by utilizing a large language model; based on an analysis result, through semantic similarity calculation and a structured reasoning mechanism, positioning a target knowledge point most related to the user intention layer by layer in a knowledge base; furthermore, a teaching video timestamp corresponding to the knowledge point is determined, and interpretable path information is generated based on a reasoning path in the positioning process. The method not only can realize accurate positioning of a key frame level, but also has traceable and explainable semantic alignment capability, and can remarkably improve intelligent understanding and positioning precision of teaching video content.

Description

Technical Field

[0001] The present invention belongs to the field of video retrieval, and in particular relates to a method for accurately locating video frames based on a hierarchical knowledge base. Background Art

[0002] With the rapid development of online education technology, particularly the widespread use of MOOC platforms, the volume of educational resources, especially instructional videos, has exploded. Against this backdrop, quickly and accurately locating the specific knowledge points of interest within this vast sea of ​​instructional videos has become a core challenge in intelligent education systems.

[0003] Existing video content retrieval methods mainly include the following three categories:

[0004] 1. Text retrieval method based on keyword matching

[0005] This type of approach typically uses subtitles, transcripts, or metadata in the video to perform keyword-level text matching. While this approach is simple to implement and offers fast retrieval, it lacks semantic understanding and struggles with semantic misalignment between questions and content, or contextual reasoning.

[0006] 2. Image similarity matching method based on visual features

[0007] By extracting image features from video frames (such as color histograms and SIFT (Scale-Invariant Feature Transform)), visual matching is achieved with query images or reference frames. This approach relies on salient visual features, has limited recognition capabilities for abstract teaching content (such as mathematical derivations and textual presentations), and cannot support localization requirements driven by natural language questions.

[0008] 3. Method based on speech recognition and timeline indexing

[0009] Automatic speech recognition (ASR) technology is used to transcribe the audio content of instructional videos and construct a timeline-based retrieval index structure, which can locate the time period of specific sentences. However, this approach is limited by speech recognition accuracy and lacks integration with the teaching knowledge structure, making it difficult to support high-level semantic reasoning and content understanding.

[0010] Although the above technologies can achieve basic video retrieval and positioning in specific scenarios, the following major problems still exist overall:

[0011] 1) Insufficient semantic understanding ability:

[0012] Most current methods are based on shallow vocabulary matching, lack an understanding of the deep semantic structure of the teaching content, and are unable to complete the knowledge association and semantic positioning of complex problems.

[0013] 2) Lack of knowledge modeling support:

[0014] Existing systems usually do not build a systematic course knowledge organization structure and lack the reasoning path from "user questions" to "knowledge content", which limits the efficient matching capabilities.

[0015] 3) Coarse positioning granularity:

[0016] Most methods can only locate the video chapter or time period level, and it is difficult to accurately locate the key frames corresponding to specific knowledge points (such as PPT pages or explanation clips).

[0017] 4) Poor interpretability:

[0018] Retrieval or recommendation results often lack clear reasoning paths or semantic support, making the results difficult to verify and interpret, affecting user trust and system application.

[0019] Therefore, there is an urgent need for a video content analysis and positioning method with deep semantic understanding, hierarchical knowledge reasoning and refined positioning capabilities to meet the urgent needs of intelligent content retrieval, precise positioning and high interpretability in complex online education platforms such as MOOCs. Summary of the Invention

[0020] In order to solve the above problems existing in the prior art, the present invention provides a method for accurately locating video frames based on a hierarchical knowledge base. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0021] A video frame precise positioning method based on a hierarchical knowledge base, comprising:

[0022] Preprocess each teaching video of a MOOC on an online education platform and extract representative key frames and their corresponding timestamps in the original video;

[0023] Based on the preprocessing results, the visual language model is used to extract key knowledge points and build a hierarchical knowledge base.

[0024] Receive natural language questions input by users and perform deep semantic analysis using a large language model;

[0025] Based on the deep semantic analysis results of the natural language question, in the knowledge base, combining semantic similarity calculation and structured reasoning mechanism, gradually locate the target knowledge points that are most relevant to the question semantics;

[0026] Determine the target timestamp corresponding to the target knowledge point in the teaching video to achieve accurate positioning, and generate interpretable path information based on the inference path in the positioning process.

[0027] In one embodiment of the present invention, the process of pre-processing each teaching video of a MOOC on an online education platform and extracting representative key frames includes:

[0028] Perform audio and video synchronization alignment for each teaching video;

[0029] For the teaching video after audio and video synchronization alignment, the video content is reasonably segmented by analyzing the breakpoints in the audio and the changes in the visual frames, and the fragments are cut to obtain the corresponding multiple video clips;

[0030] For the multiple video clips obtained from the teaching video, representative key frames are extracted based on inter-frame similarity analysis; the key frames represent the scenes with the most concentrated knowledge content in the teaching video.

[0031] In one embodiment of the present invention, the process of extracting the timestamp corresponding to the representative key frame in the original video includes:

[0032] For each key frame, the timestamp of the key frame in the corresponding teaching video is uniquely identified as the corresponding original video timestamp.

[0033] In one embodiment of the present invention, extracting key knowledge points based on the preprocessing results with the aid of a large visual language model and constructing a hierarchical knowledge base includes:

[0034] Using a large visual language model, semantic analysis is performed on the key frames extracted from the multiple teaching videos and the speech content of the corresponding video clips to extract multiple relevant knowledge points;

[0035] According to the hierarchical relationship between related knowledge points, a top-down hierarchical knowledge base is constructed. The knowledge base has a tree structure, which reflects the hierarchical correspondence between courses, units, sections, key frames and knowledge points.

[0036] In one embodiment of the present invention, the process of performing deep semantic parsing of natural language questions using a large language model includes:

[0037] Using a large language model, the knowledge points and intent types in the natural language questions are identified and extracted, and converted into standardized language representations that match the knowledge base as deep semantic parsing results.

[0038] In one embodiment of the present invention, the deep semantic parsing result is expressed as:

[0039] S=φ LLM (Q) = {K, I};

[0040] Among them, Q is the natural language question input by the user, φ LLM (·) is the deep semantic parsing function implemented by the large language model; S is the deep semantic parsing result; K is the knowledge point, and I is the intent type.

[0041] In one embodiment of the present invention, based on the deep semantic analysis results of the natural language question, the target knowledge points most relevant to the question semantics are gradually located in the knowledge base by combining semantic similarity calculation and structured reasoning mechanism. The formula used is expressed as follows:

[0042] k * =argmax[Sim(S,R(k i ))+λ·ψ(k i ,H,S)],i=0,...,n;

[0043] Among them, S is the deep semantic analysis result; k i is the knowledge point with index i in the knowledge base, n is the total number of knowledge points in the knowledge base; R(k i ) is the knowledge point k i The semantic representation of the text vector; Sim(,) is the cosine similarity calculation function of the text vector; ψ(k i ,H,S) is a pre-designed structured reasoning score function, which is used to reflect the reasoning priority of the knowledge point with index i in the structural hierarchy; λ is the weighting coefficient; k * Target knowledge points.

[0044] In one embodiment of the present invention, the expression of the pre-designed structured reasoning score function is:

[0045]

[0046] Among them, Unit(k i ) is the knowledge point k i The semantic vector of the unit to which it belongs; Lesson(k i ) is the knowledge point k i The semantic vector of the section; Sim(S,Unit(k i )) and Sim(S,Lesson(k i )) is the deep semantic analysis result S and knowledge point k i The semantic similarity between the upper unit and the upper section, Contiguity(k i ) is used as the scoring basis for structural adjacency; α1, α2, and α3 are the corresponding weight coefficients;

[0047]

[0048] Among them, Z is the number of adjacent knowledge points; k'∈N(k i ) is the same as knowledge point k i A set of adjacent knowledge points belonging to the same upper-level node.

[0049] In one embodiment of the present invention, determining the target timestamp corresponding to the target knowledge point in the teaching video includes:

[0050] According to the mapping relationship between the knowledge points and the teaching video timestamps established in the knowledge base construction process, the time position corresponding to the target knowledge point in the teaching video is retrieved and determined as the target timestamp.

[0051] In one embodiment of the present invention, generating interpretable path information based on the inference path in the positioning process includes:

[0052] According to the knowledge base and the target timestamp, the system traces back to the root node of the knowledge base layer by layer from bottom to top, records the target knowledge points, reasoning paths and semantic similarity scores of each level hit during the positioning process, and forms complete interpretable path information.

[0053] Beneficial effects of the present invention:

[0054] In the video frame precise positioning method based on the hierarchical knowledge base provided by the embodiment of the present invention, first, each teaching video of a MOOC in the online education platform is preprocessed to extract representative key frames and their corresponding timestamps in the original video; secondly, based on the preprocessing results, the key knowledge points are extracted with the help of a large visual language model, and a hierarchical knowledge base is constructed; next, the natural language questions input by the user are received, and the large language model is used to perform deep semantic analysis on them; then, based on the deep semantic analysis results of the natural language questions, the target knowledge points most relevant to the question semantics are gradually located in the knowledge base by combining semantic similarity calculation and structured reasoning mechanism; finally, the target timestamp corresponding to the target knowledge point in the teaching video is determined, thereby achieving precise positioning, and generating interpretable path information based on the reasoning path in the positioning process.

[0055] With the help of a large language model, this invention integrates strategies such as semantic matching, hierarchical linkage reasoning and same-level logical deduction to construct a complete reasoning chain from "user questions" to "knowledge nodes" and then to "video key frames", ultimately achieving precise timestamp positioning at the key frame level, and providing a traceable and explainable semantic alignment process, which can significantly improve the intelligent understanding and positioning accuracy of teaching video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of a process for accurately locating video frames based on a hierarchical knowledge base provided by an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of the structure of the knowledge base constructed according to an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of the process of understanding the method for accurately locating video frames based on a hierarchical knowledge base provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0060] In response to the prominent problems of existing teaching video content retrieval technology in terms of insufficient semantic understanding ability, chaotic knowledge organization structure, coarse positioning granularity and poor result interpretability, the core goal of this invention is to provide a video frame precise positioning method based on a hierarchical knowledge base, so as to achieve deep semantic association modeling and frame-level precise positioning between user natural language questions and teaching video content.

[0061] Specifically, the technical problems to be solved by the present invention include:

[0062] How to achieve deep semantic parsing of user natural language questions and extract semantic representations that can be used for knowledge base reasoning and matching;

[0063] How to accurately locate the knowledge points that reflect the user's question intent and further associate them with the corresponding key video frames through semantic similarity calculation, upper and lower-level linkage, and same-level reasoning mechanisms in a multi-level knowledge base structure with semantic association and logical reasoning capabilities?

[0064] How to provide a traceable explanation path for positioning results to enhance the system's comprehensibility, credibility, and scalability in teaching scenarios;

[0065] How to achieve efficient matching and real-time response capabilities for large-scale teaching video content while ensuring high positioning accuracy.

[0066] Through the above technical path, the present invention aims to break through the "surface matching" limitations of traditional methods on video content, construct a "knowledge-mediated, reasoning-based" semantic positioning mechanism for teaching videos, and realize a high-precision, strong semantic, and explainable intelligent matching and positioning process from natural language questions to teaching key frames.

[0067] In order to achieve the above-mentioned purpose, the embodiment of the present invention provides a method for accurately locating video frames based on a hierarchical knowledge base. Figure 1 As shown, the method may include the following steps:

[0068] S1, pre-processes each teaching video of a MOOC on the online education platform and extracts representative key frames and their corresponding timestamps in the original video;

[0069] On current online education platforms, a Massive Open Online Course (MOOC) typically consists of multiple instructional videos, primarily consisting of PowerPoint presentations and teacher explanations. These videos cover various key points of the course, and a course is typically organized into a unit-section-video structure. For example, an "Introduction to Artificial Intelligence" course might be divided into units such as "Basics," "Machine Learning," and "Deep Learning." Each unit is further divided into several sections, each corresponding to one or more instructional videos.

[0070] In an embodiment of the present invention, the process of preprocessing each teaching video of a MOOC on an online education platform to extract representative key frames includes:

[0071] 1) Perform audio and video synchronization alignment for each teaching video;

[0072] When producing MOOC instructional videos, the recording process may be disconnected. For example, the instructor's audio is typically recorded via a microphone, while the PowerPoint presentations are primarily captured via screen capture or camera. The audio and video are captured via different devices or software channels. Even on the same computer, there may be slight discrepancies between the internal clocks of the screen recording software and the microphone audio input. Alternatively, delays may be introduced during post-processing of the instructional video, such as due to improper transcoding or compression. Furthermore, the playback of instructional videos may experience audio and video desynchronization due to factors such as network fluctuations and device performance.

[0073] To accurately convey teaching information and ensure learning outcomes, audio and video synchronization is essential. Existing processing software or algorithms can be used to perform audio and video synchronization on each teaching video, ensuring accurate synchronization of audio and video. For example, synchronizing the teacher's lip movements with their voice, synchronizing PPT actions with their explanations, and so on.

[0074] 2) For the teaching video after audio and video synchronization alignment, the video content is reasonably segmented by analyzing the breakpoints in the audio and the changes in the visual frames, and the fragments are cut to obtain the corresponding multiple video clips;

[0075] For each teaching video after the audio and video synchronization alignment processing, segmentation processing is also required.

[0076] Specifically, for a teaching video that has undergone audio and video synchronization, by analyzing the breakpoints in the audio and the changes in the visual frames, it is possible to find the gaps in the audio, that is, the end points or intervals of the explanation content, and at the same time, find the completion points of the knowledge points presented in the PPT content of the teaching video. For example, for a certain knowledge point, the time when the teacher's audio explanation is completed and the relevant sentences on the PPT page are complete can be used as a segmentation point. By finding multiple such segmentation points, the teaching video that has undergone audio and video synchronization can be segmented to obtain multiple corresponding video segments.

[0077] The search and segmentation can be realized by using existing video processing algorithms or related software, which will not be described in detail here.

[0078] Through segment cutting, each teaching video after audio and video synchronization alignment can be processed and segmented according to knowledge points, thus providing a basis for the subsequent construction of the knowledge base.

[0079] 3) Extracting representative key frames from multiple video clips obtained from the teaching video based on inter-frame similarity analysis;

[0080] For all the video clips obtained from a teaching video, key frames can be extracted through inter-frame similarity analysis. Key frames represent the scenes with the most concentrated knowledge content in the teaching video. For example, when explaining a knowledge point, relevant content will appear on the PPT page one after another. After the content of the current PPT page is fully displayed and the explanation is completed, the teacher will turn the page to explain the next page of PPT. Then, before turning the page, the frame corresponding to the moment when the current PPT page is fully displayed and the explanation is completed will be used as a key frame.

[0081] Therefore, inter-frame similarity analysis mainly involves frame-by-frame identification, finding similar content frame by frame until a frame with a large difference in content from the next frame. This can be achieved using existing video processing algorithms, which will not be described in detail here.

[0082] After identifying the key frames, it is necessary to identify the original video timestamps of each key frame. Specifically, the process of extracting the timestamps corresponding to representative key frames in the original video is to uniquely identify the timestamp of each key frame in the corresponding teaching video as the corresponding original video timestamp.

[0083] The original video timestamp is marked for each key frame in order to trace back the corresponding timestamp information and achieve positioning.

[0084] The present invention provides a key frame extraction and timestamp binding mechanism, which specifically combines audio and video alignment, visual frame changes and semantic analysis to extract representative frames as "knowledge anchors" and record their timestamps in the original video to achieve accurate frame-level positioning capabilities.

[0085] S2, based on the preprocessing results, uses the visual language model to extract key knowledge points and build a hierarchical knowledge base;

[0086] Through the processing of S1, each teaching video is pre-processed to obtain corresponding multiple video clips, and the key frames are identified and the original video timestamps are marked. These are the pre-processing results of the multiple teaching videos.

[0087] On this basis, we use the visual language model to extract knowledge points and build a hierarchical knowledge base. The specific steps include:

[0088] S21, using a large visual language model, performing semantic analysis on the key frames extracted from the multiple teaching videos and the speech content of the corresponding video clips to extract multiple relevant knowledge points;

[0089] S22, constructing a top-down hierarchical knowledge base based on the hierarchical relationship between related knowledge points, wherein the knowledge base has a tree-like structure, reflecting the hierarchical correspondence between courses, units, sections, key frames and knowledge points.

[0090] The embodiment of the present invention can adopt any existing large visual language model, and rely on the performance of the large visual language model to perform semantic extraction on the key frames extracted from the multiple teaching videos and the voice content of the corresponding video clips, and extract multiple relevant knowledge points contained in the corresponding MOOC. Since the knowledge points have a hierarchical relationship between superiors and subordinates, a hierarchical knowledge system can be constructed based on this, forming a top-down "course-unit-section-keyframe-knowledge point" tree-shaped knowledge base structure, that is, a knowledge base of a hierarchical and inferential knowledge base is constructed. This knowledge base can clearly reflect the hierarchical correspondence between courses, units, sections, keyframes and knowledge points, providing an organizational basis and semantic support for subsequent semantic understanding and frame-level positioning.

[0091] The hierarchical structure of the constructed knowledge base can be found in Figure 2 Understand, where Mooc means Massive Open Online Courses, i.e. course, Unit means unit, Lesson means section, and Content means the content of the key frame, in which various knowledge points can be divided from top to bottom.

[0092] S3 receives natural language questions input by users and uses a large language model to perform deep semantic analysis;

[0093] S3 represents the semantic analysis of user questions. The user can be a student, etc.

[0094] During the frame-level localization phase, the system first receives a natural language question from the user and performs deep semantic analysis using a large language model. A natural language question is typically a text entry expressing the user's question. The large language model (LLM) can be any existing large language model.

[0095] Specifically, the process of using a large language model to perform deep semantic analysis of natural language questions includes:

[0096] Using a large language model, the knowledge points and intent types in the natural language questions are identified and extracted, and converted into standardized language representations that match the knowledge base as deep semantic parsing results.

[0097] The embodiment of the present invention utilizes the semantic parsing capability of a large language model for natural language questions to achieve the above-mentioned processing.

[0098] The deep semantic analysis result is expressed as:

[0099] S=φ LLM (Q) = {K, I};

[0100] Among them, Q is the natural language question input by the user, φ LLM (·) is a deep semantic parsing function implemented by a large language model, which includes steps such as knowledge point extraction and intent recognition; S is the result of deep semantic parsing; K is the knowledge point, and I is the intent type.

[0101] For example, in response to a user's natural language question, "When was the Dartmouth Conference held?", the large language model analyzes and identifies the knowledge points "Dartmouth Conference" and "holding time," corresponding to the intent type "knowledge location - Dartmouth Conference holding time." In reality, intent types can encompass multiple categories, such as "knowledge location," "technical Q&A," or "casual chat."

[0102] This invention introduces a large language model, leveraging its capabilities to support deep semantic parsing of user-entered natural language questions. By identifying intent, extracting keywords, and transforming semantic structures, it generates a semantic representation suitable for knowledge base matching. Unlike traditional techniques that rely heavily on keyword matching or shallow intent recognition, this invention can identify implicit intent, core concepts, and relevant context within complex questions, significantly improving the accuracy of matching user questions with knowledge content and meeting diverse and personalized teaching query needs.

[0103] S4, based on the deep semantic analysis results of the natural language question, gradually locate the target knowledge points that are most relevant to the question semantics in the knowledge base by combining semantic similarity calculation and structured reasoning mechanism;

[0104] S4 can be understood as knowledge base semantic matching and reasoning retrieval. The formula used is:

[0105] k * =argmax[Sim(S,R(k i ))+λ·ψ(k i ,H,S)],i=0,...,n;

[0106] Among them, S is the deep semantic analysis result; k i is the knowledge point with index i in the knowledge base, n is the total number of knowledge points in the knowledge base, which can be numbered and statistically determined when constructing the knowledge base; R(k i ) is the knowledge point k i The semantic representation is obtained by using existing technology to obtain semantic vectors; Sim(,) is the cosine similarity calculation function of text vectors; ψ(k i ,H,S) is a pre-designed structured reasoning score function, which is used to reflect the reasoning priority of the knowledge point with index i in the structural hierarchy; λ is the weighting coefficient; k * Target knowledge points.

[0107] The currently designed structured reasoning score function mainly solves the problems of inaccurate semantic matching of knowledge points and isolated semantic information. When users ask vague or generalized natural language questions, it is often difficult to achieve accurate matching by simply relying on the semantic vector of the knowledge point itself. To this end, the present invention expands the structured reasoning score function from two key dimensions: first, by calculating the semantic consistency (structural consistency) between the user's question and the unit and section to which the knowledge point belongs, the semantic scope is effectively limited to avoid hallucination matching; second, considering the correlation between the knowledge point and the adjacent nodes in the tree structure (structural adjacency), the local semantic density is enhanced and the context understanding ability is improved;

[0108] For example, when faced with the user's question "What is the derivation formula for backpropagation?", the system may simultaneously retrieve multiple knowledge points such as "the concept of backpropagation," "the process of backpropagation," and even "the formula for gradient descent." However, only the node "Backpropagation > Formula" is both highly semantically relevant and structurally consistent with its unit "Backpropagation" and section "Neural Network Training." Its adjacent nodes, such as "Backpropagation > Process" and "Backpropagation > Error Calculation," are also closely related to the question semantics. Therefore, this knowledge point has the highest combined score in structural consistency and structural adjacency, and is more likely to be judged as the optimal match.

[0109] In order to achieve the above two points, the structured reasoning score function ψ(k i ,H,S) is divided into multiple sub-items, so the expression of the pre-designed structured reasoning score function is:

[0110]

[0111] Among them, Unit(k i ) is the knowledge point k i The semantic vector of the unit to which it belongs; Lesson(k i ) is the knowledge point k i The semantic vector of the section; Sim(S,Unit(k i )) and Sim(S,Lesson(k i )) is the deep semantic analysis result S and knowledge point k i The semantic similarity between the upper unit and the upper section; Contiguity (k i ) is used as the scoring basis for structural adjacency; α1, α2, and α3 are the corresponding weight coefficients;

[0112]

[0113] Among them, Z is the number of adjacent knowledge points; k'∈N(k i ) is the same as knowledge point k i The set of adjacent knowledge points belonging to the same upper-level node. This formula measures the semantic proximity between the user question S and the "surrounding neighbor knowledge points" in the structure.

[0114] The Sim(S,R(k i )), namely the text vector cosine similarity calculation function Sim(,), is one of the core components of the entire system, used to measure the semantic representation of the problem generated by the large language model S = φ LLM (Q) is related to the semantic representation R(k) of the knowledge point.

[0115]

[0116] Among them, S is the semantic vector (embedding vector) generated by the large language model for the user's natural language question, and R(k) is the semantic vector of the semantic representation of knowledge point k by the large language model.

[0117] It is understandable that the process of locating target knowledge points based on natural language questions raised by users is a process of reasoning and retrieval, which requires traversing the hierarchical knowledge base and calculating semantic relevance until the knowledge point with the highest degree of relevance to the user's question is found as the target knowledge point.

[0118] The present invention provides a structured reasoning mechanism and a multi-layer semantic matching function. Specifically:

[0119] Based on the knowledge base structure, a structured reasoning score function is designed to perform deep semantic matching between user questions and candidate knowledge points. In the knowledge point matching process, a structured semantic reasoning mechanism is introduced, which not only considers the semantic similarity of the knowledge point itself, but also introduces the semantic consistency of the upper structure of the unit and the subsection to which it belongs, as well as the structural relevance of the adjacent knowledge points under the same subsection, and constructs a multi-layer semantic fusion scoring mechanism. Compared with the traditional method of matching based only on the similarity of "question and knowledge point itself", this formula integrates the semantic consistency of the upper structure of the knowledge point (unit, subsection) and the horizontal structural relevance (adjacent knowledge point), reflecting the advantages of the present invention in multi-layer semantic relationship modeling. It can not only match the answer to the question with the closest semantics, but also perform semantic completion and error correction with the help of contextual information and structural adjacency, effectively alleviating the matching difficulties caused by the ambiguity of user expression or the change of knowledge point definition, and greatly improving the accuracy and robustness of knowledge matching. This structure-enhanced reasoning mechanism is an important support for the system to achieve frame-level precise positioning, and is also a key innovation for achieving high-quality intelligent retrieval in knowledge-intensive video content. It can improve the accuracy and robustness of knowledge point positioning and realize refined semantic reasoning capabilities for complex structured knowledge systems.

[0120] S5, determining the target timestamp corresponding to the target knowledge point in the teaching video, thereby achieving accurate positioning, and generating interpretable path information based on the inference path in the positioning process.

[0121] Determining the target timestamp corresponding to the target knowledge point in the teaching video includes:

[0122] According to the mapping relationship between the knowledge points and the teaching video timestamps established in the knowledge base construction process, the time position corresponding to the target knowledge point in the teaching video is retrieved and determined as the target timestamp.

[0123] When constructing a knowledge base, the present invention not only arranges and numbers each knowledge point in a hierarchical manner, but also associates each knowledge point with the corresponding instructional video by timestamp, identifying the timestamp corresponding to the instructional video. Specifically, each knowledge point is associated with a corresponding PPT keyframe image. Based on the timestamps associated with all knowledge points, a mapping relationship between each knowledge point and the instructional video can be constructed, for example, in a table format.

[0124] Therefore, after completing the semantic parsing of the user question and the inference retrieval of the knowledge base, we can map the timestamps of the located target knowledge points in the original teaching video to the timestamps. This will yield the target timestamp, which is the original video timestamp uniquely identified and recorded during the video structure extraction phase. This timestamp is then used as the output of the system.

[0125] Traditional methods can only locate videos at the chapter or segment level, making them difficult to meet the refined requirements of frame-level content search. This invention preprocesses the video structure, identifies and records keyframe images and their precise timestamps, and combines this with semantic reasoning to achieve second-level location of keyframes in instructional videos, meeting the high standards of teaching systems for accurate retrieval and content alignment.

[0126] At the same time, in order to ensure the interpretability and verifiability of the frame positioning results, interpretable path information can also be generated based on the inference path during the positioning process, including:

[0127] According to the knowledge base and the target timestamp, the system traces back to the root node of the knowledge base layer by layer from bottom to top, records the target knowledge points, reasoning paths and semantic similarity scores of each level hit during the positioning process, and forms complete interpretable path information.

[0128] It can be understood that the target timestamp, that is, the target knowledge point, corresponds to a node in the knowledge base. Therefore, the tree branch corresponding to the target knowledge point, that is, the "course-unit-section-keyframe-knowledge point" branch to which it belongs, can be determined as the reasoning path. At the same time, the similarity score obtained when calculating the semantic similarity in the process of locating the target knowledge point can be output together with the target knowledge point as interpretable path information and displayed to users and other relevant personnel, thereby providing interpretability.

[0129] The timestamp positioning mechanism provided by the present invention ensures the precise correspondence from semantic questions to video time points, and is a key link in realizing semantic-driven video frame-level retrieval.

[0130] The main concept of the method of the embodiment of the present invention can be found in Figure 3 As shown above, unlike traditional retrieval methods that rely solely on keywords or visual features, this method leverages the semantic understanding capabilities of large models and a hierarchical search strategy within the knowledge base to significantly improve the accuracy and interpretability of knowledge location. It can be applied to MOOC platforms, online education systems, and knowledge-based teaching videos for frame-level location and intelligent analysis of specific knowledge points.

[0131] This paper aims to address the technical challenges of existing instructional video retrieval technologies, including insufficient semantic understanding, low positioning accuracy, a lack of a knowledge support system, and unexplainable results. By proposing a method for accurately locating video frames based on a hierarchical knowledge base, this method leverages a multi-level knowledge base with semantic association and reasoning capabilities. This method, tailored to user natural language questions, enables precise identification and timestamp location of knowledge content within instructional videos.

[0132] Specifically, this invention leverages a large language model to abstractly model video content, forming a hierarchical knowledge base that reflects the relationship between instructional semantics and knowledge organization, providing structured semantic support for subsequent location-based learning. This knowledge base, built on the course content, establishes a multi-layered knowledge representation, boasting robust semantic support and reasoning capabilities. This effectively supports semantic alignment and precise retrieval between natural language questions and video content.

[0133] During the actual positioning process, the system receives natural language questions posed by users, extracts user intent and keywords through semantic parsing, and performs semantic matching and hierarchical retrieval within the hierarchical knowledge base. The positioning algorithm comprehensively considers factors such as semantic similarity, knowledge structure paths, and the credibility of reasoning links to accurately identify video clips highly relevant to the question, map them to corresponding video frames in the instructional video, and obtain frame-level timestamp information.

[0134] To improve interpretability, the present invention also provides an explanation of the positioning path, including the knowledge points and course retrieval links corresponding to the selected frame, to ensure that the positioning results are transparent and credible.

[0135] It can be seen that the method of the present invention integrates strategies such as semantic matching, hierarchical linkage reasoning and same-level logical deduction, constructs a complete reasoning chain from "user questions" to "knowledge nodes" and then to "video key frames", and finally achieves precise timestamp positioning at the key frame level, and provides a traceable and explainable semantic alignment process, which can significantly improve the intelligent understanding and positioning accuracy of teaching video content.

[0136] It should be noted that, in the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0137] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0138] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0139] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A video frame accurate positioning method based on a hierarchical knowledge base, characterized in that: include: Preprocess each teaching video of a MOOC on an online education platform and extract representative key frames and their corresponding timestamps in the original video; Based on the preprocessing results, the visual language model is used to extract key knowledge points and build a hierarchical knowledge base. Receive natural language questions input by users and perform deep semantic analysis using a large language model; Based on the deep semantic analysis results of the natural language question, in the knowledge base, combining semantic similarity calculation and structured reasoning mechanism, gradually locate the target knowledge points that are most relevant to the question semantics; Determine the target timestamp corresponding to the target knowledge point in the teaching video to achieve accurate positioning, and generate interpretable path information based on the inference path in the positioning process.

2. The method according to claim 1, characterized in that The process of preprocessing each teaching video of a MOOC on an online education platform and extracting representative key frames includes: Perform audio and video synchronization alignment for each teaching video; For the teaching video after audio and video synchronization alignment, the video content is reasonably segmented by analyzing the breakpoints in the audio and the changes in the visual frames, and the fragments are cut to obtain the corresponding multiple video clips; For the multiple video clips obtained from the teaching video, representative key frames are extracted based on inter-frame similarity analysis; the key frames represent the scenes with the most concentrated knowledge content in the teaching video.

3. The method according to claim 2, characterized in that The process of extracting the timestamps corresponding to representative key frames in the original video includes: For each key frame, the timestamp of the key frame in the corresponding teaching video is uniquely identified as the corresponding original video timestamp.

4. The method according to claim 3, characterized in that Based on the preprocessing results, the visual language model is used to extract key knowledge points and build a hierarchical knowledge base, including: Using a large visual language model, semantic analysis is performed on the key frames extracted from the multiple teaching videos and the speech content of the corresponding video clips to extract multiple relevant knowledge points; According to the hierarchical relationship between related knowledge points, a top-down hierarchical knowledge base is constructed. The knowledge base has a tree structure, which reflects the hierarchical correspondence between courses, units, sections, key frames and knowledge points.

5. The method according to claim 4, characterized in that The process of using a large language model to perform deep semantic analysis of natural language questions includes: Using a large language model, the knowledge points and intent types in the natural language questions are identified and extracted, and converted into standardized language representations that match the knowledge base as deep semantic parsing results.

6. The method according to claim 5, characterized in that The deep semantic analysis result is expressed as: S=φ LLM (Q)={K,I}; Among them, Q is the natural language question input by the user, φ LLM (·) is the deep semantic parsing function implemented by the large language model; S is the deep semantic parsing result; K is the knowledge point, and I is the intent type.

7. The method according to claim 1 or 6, characterized in that Based on the deep semantic analysis results of the natural language question, in the knowledge base, the target knowledge points most relevant to the question semantics are gradually located by combining semantic similarity calculation and structured reasoning mechanism. The formula used is expressed as follows: k * =argmax[Sim(S,R(k i ))+λ·ψ(k i ,H,S)],i=0,...,n; Among them, S is the deep semantic analysis result; k i is the knowledge point with index i in the knowledge base, n is the total number of knowledge points in the knowledge base; R(k i ) is the knowledge point k i The semantic representation of the text vector; Sim(,) is the cosine similarity calculation function of the text vector; ψ(k i ,H,S) is a pre-designed structured reasoning score function, which is used to reflect the reasoning priority of the knowledge point with index i in the structural hierarchy; λ is the weighting coefficient; k * Target knowledge points.

8. The method according to claim 7, characterized in that The expression of the pre-designed structured reasoning score function is: Among them, Unit(k i ) is the knowledge point k i The semantic vector of the unit to which it belongs; Lesson(k i ) is the knowledge point k i The semantic vector of the section; Sim(S,Unit(k i )) and Sim(S,Lesson(k i )) is the deep semantic analysis result S and knowledge point k i The semantic similarity between the upper unit and the upper section, Contiguity(k i ) is used as the scoring basis for structural adjacency; α1, α2, and α3 are the corresponding weight coefficients; Among them, Z is the number of adjacent knowledge points; k'∈N(k i ) is the same as knowledge point k i A set of adjacent knowledge points belonging to the same upper-level node.

9. The method according to claim 1, characterized in that Determining the target timestamp corresponding to the target knowledge point in the teaching video includes: According to the mapping relationship between the knowledge points and the teaching video timestamps established in the knowledge base construction process, the time position corresponding to the target knowledge point in the teaching video is retrieved and determined as the target timestamp.

10. The method according to claim 8, characterized in that The generating of interpretable path information based on the inference path in the positioning process includes: According to the knowledge base and the target timestamp, the system traces back to the root node of the knowledge base layer by layer from bottom to top, records the target knowledge points, reasoning paths and semantic similarity scores of each level hit during the positioning process, and forms complete interpretable path information.

Images