Method and device for knowledge positioning of video based on large model, storage medium and computer device

Abstract

The application discloses a method and device for knowledge positioning of a video based on a large model, a storage medium and computer equipment, and the method comprises the following steps: acquiring video data, extracting the semantic feature vector of each video frame and the speech text sequence containing timestamp information, and performing time alignment processing to generate a multi-modal time sequence; the multi-modal time sequence is divided to obtain a plurality of segments, each segment is input into a first pre-training large model, knowledge point description information is output, an inverted index database is constructed by taking the knowledge point description information as an index key and the corresponding time interval as an index value; in response to natural language query content input by a user, the natural language query content is converted into query semantic features, semantic correlation calculation is performed in the inverted index database according to the query semantic features, at least one matched knowledge point description information and the corresponding candidate time interval are recalled, a target time interval is obtained according to the at least one matched knowledge point description information and the corresponding candidate time interval, and a target video positioning interval covering the natural language query content is cut from the video data according to the target time interval.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a method, apparatus, storage medium, and computer equipment for knowledge localization of video based on a large model. Background Technology

[0002] With the explosive growth of knowledge-based video resources such as online education, corporate training, and academic lectures, how to help users quickly and accurately locate the knowledge points they need from massive amounts of long video content has become a key technical problem that urgently needs to be solved in the field of video information retrieval. Unlike short entertainment videos, knowledge-based videos are usually high in content density, with explanations primarily in audio format and synchronized with video, and the overall length is often more than 30 minutes, with some courses even reaching 1-2 hours. When faced with such long videos, the specific knowledge points that users need are often scattered throughout the lengthy video content, making location and extraction extremely costly. Traditional solutions mainly rely on manually annotated video titles, descriptions, or chapter tags for retrieval. Users search for videos by keyword matching and then manually drag the progress bar to repeatedly browse within the video to find the target content. This metadata-based retrieval method is extremely coarse-grained, only able to locate the video file itself or a rough chapter level, failing to meet users' needs for precise time and location within the video, resulting in users spending a lot of time in the search process and low learning efficiency.

[0003] To achieve the retrieval and location of content within videos, several analysis methods based on the video content itself have been developed in existing technologies. One method is keyword retrieval based on Automatic Speech Recognition (ASR) subtitles. The system transcribes the audio content in the video into timestamped subtitle text, and the user inputs keywords for string matching to locate the first time the keyword appears in the subtitle. Another method is retrieval based on video keyframe analysis. This method extracts keyframes from the video using computer vision technology and assigns labels to the keyframes using image classification or object detection models, thereby understanding the visual content of the video.

[0004] However, the aforementioned existing technical solutions still have significant shortcomings in achieving accurate knowledge localization. Keyword retrieval based on ASR captions can only perform literal matching and cannot understand synonyms, near-synonyms, or contextual semantics. For example, when a user searches for "how to apply for a loan," it cannot match caption segments explaining the "loan process." Furthermore, the accuracy of ASR captions is inherently limited and cannot cover purely visual knowledge points such as charts, demonstrations, or interface operations. The annotation capabilities based on keyframe analysis are extremely limited, typically only able to identify a limited set of pre-defined categories of objects or scenes, and unable to answer complex, abstract user natural language questions. Summary of the Invention

[0005] In view of this, this application provides a method, apparatus, storage medium, and computer device for knowledge localization in videos based on a large model. By extracting semantic feature vectors and speech text sequences to generate multimodal time-series sequences, combining the large model to generate knowledge point description information and constructing an inverted index library, and converting user natural language queries into query semantic features for semantic relevance calculation and combining time interval fusion and extraction, this solves the technical problems of existing technologies that can only perform literal keyword matching and cannot understand synonyms, near-synonyms, and contextual semantics, cannot cover purely visual knowledge points such as charts and demonstration actions, cannot answer complex and abstract user natural language questions, and cannot perform fine-grained temporal localization of knowledge points within videos.

[0006] According to one aspect of this application, a method for knowledge localization of videos based on a large model is provided, comprising: The original video data is acquired, and the visual and audio modalities of the video data are parsed respectively. The semantic feature vectors of each video frame and the speech text sequence containing timestamp information are extracted. The semantic feature vectors and the speech text sequence are aligned on the time axis to generate a multimodal time series sequence. The multimodal time series is slidably segmented according to a preset time granularity to obtain multiple segments. Each segment is input into a first pre-trained large model. The first pre-trained large model outputs at least one knowledge point description information to represent the core semantics of the current segment. An inverted index library is constructed using the knowledge point description information as the index key and the corresponding time interval as the index value. In response to a user's input natural language query, the natural language query is converted into corresponding query semantic features. Based on the query semantic features, semantic relevance is calculated in the inverted index database. At least one knowledge point description information matching the natural language query and its corresponding candidate time interval are recalled. Semantic fusion and redundancy removal operations are performed on all recalled candidate time intervals to obtain a target time interval. Based on the target time interval, a target video positioning interval covering the natural language query is extracted from the video data.

[0007] According to another aspect of this application, an apparatus for knowledge localization of video based on a large model is provided, comprising: The sequence generation module is used to acquire the original video data, parse the visual modality and audio modality of the video data respectively, extract the semantic feature vector of each video frame and the speech text sequence containing timestamp information, and align the semantic feature vector and the speech text sequence on the time axis to generate a multimodal time sequence. The index library construction module is used to slide and segment the multimodal time series according to a preset time granularity to obtain multiple segments. Each segment is input into the first pre-trained large model. The first pre-trained large model outputs at least one knowledge point description information to represent the core semantics of the current segment. The inverted index library is constructed using the knowledge point description information as the index key and the corresponding time interval as the index value. The video capture module is used to respond to the natural language query content input by the user, convert the natural language query content into corresponding query semantic features, perform semantic relevance calculation in the inverted index database based on the query semantic features, recall at least one knowledge point description information and its corresponding candidate time interval that match the natural language query content, and perform semantic fusion and redundancy removal operations on all recalled candidate time intervals to obtain the target time interval. Based on the target time interval, the module captures the target video positioning interval that covers the natural language query content from the video data.

[0008] According to another aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described method for knowledge localization of video based on a large model.

[0009] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described method for knowledge localization of video based on a large model.

[0010] By employing the aforementioned technical solutions, this application provides a method, apparatus, storage medium, and computer device for knowledge localization in videos based on a large model. By extracting semantic feature vectors and speech-text sequences to generate multimodal time-series sequences, the system simultaneously possesses the ability to perceive both visual and speech content, overcoming the limitation of pure ASR subtitle retrieval in failing to cover purely visual knowledge points such as charts and demonstration actions. Through the output of knowledge point description information from a first pre-trained large model and the construction of an inverted index library, semantic-level matching is achieved, overcoming the limitation of traditional keyword retrieval which can only perform literal matching and cannot understand synonyms, near-synonyms, and contextual semantics. By transforming user natural language queries into query semantic features and performing semantic relevance calculations, the limitation of keyframe analysis, which can only identify preset categories of objects or scenes and cannot answer complex and abstract questions, is overcome. By performing semantic fusion and redundancy removal operations on candidate time intervals, scattered related knowledge point fragments are integrated into a complete target video localization interval, reducing the time spent by users searching for specific knowledge points in long videos.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 The illustration shows a flowchart of a method for knowledge localization of video based on a large model, according to an embodiment of this application. Figure 2 This illustration shows a schematic diagram of a device for knowledge localization of video based on a large model, according to an embodiment of this application. Figure 3 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0013] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0014] This embodiment provides a method for knowledge localization in videos based on a large model, such as... Figure 1 As shown, the method includes: Step 101: Obtain the original video data, parse the visual modality and audio modality of the video data respectively, extract the semantic feature vector of each video frame and the speech text sequence containing timestamp information, and align the semantic feature vector and the speech text sequence on the time axis to generate a multimodal time sequence.

[0015] Step 102: The multimodal time series is slidably segmented according to a preset time granularity to obtain multiple segments. Each segment is input into the first pre-trained large model. The first pre-trained large model outputs at least one knowledge point description information to represent the core semantics of the current segment. An inverted index library is constructed using the knowledge point description information as the index key and the corresponding time interval as the index value.

[0016] Step 103: In response to the natural language query content input by the user, the natural language query content is converted into corresponding query semantic features. Based on the query semantic features, semantic relevance is calculated in the inverted index database. At least one knowledge point description information matching the natural language query content and its corresponding candidate time interval are recalled. Semantic fusion and redundancy removal operations are performed on all recalled candidate time intervals to obtain the target time interval. Based on the target time interval, the target video positioning interval covering the natural language query content is extracted from the video data.

[0017] This application provides a method for knowledge localization of video based on a large model. First, the visual and audio channels of the original video data are analyzed independently: For the visual channel, a visual coding model is used to convert each frame of video into a high-dimensional semantic vector, resulting in a semantic feature vector, which is a mathematical representation of the semantic information of the image; for the audio channel, automatic speech recognition technology is used to convert the speech signal into a time-stamped speech-text sequence. In a specific embodiment, visual stream processing involves extracting keyframes of the video at a frequency of 1 frame per second (configurable), and using a pre-trained visual model (such as CLIP) to convert each frame into a high-dimensional semantic vector; auditory / text stream processing involves using ASR technology to transcribe the video audio into time-stamped subtitle text (each subtitle includes start time, end time, and text content). Subsequently, using time as the reference axis, the semantic feature vectors at the same moment are aligned and associated with the speech-text sequence to form a multimodal temporal sequence that integrates visual and audio modal information, thereby transforming the original unstructured video data into a structured data form that can be processed by the computing system.

[0018] Based on the aligned multimodal temporal sequence, an index structure supporting semantic-level retrieval is established. Specifically, the multimodal temporal sequence is slidably segmented according to a preset temporal granularity (e.g., time window length), dividing the entire video into multiple temporally continuous and partially overlapping segment units. Each segment simultaneously contains semantic feature vectors and speech-text sequences within that time period. Each segment is input into a first pre-trained large model with multimodal understanding capabilities. This model can simultaneously perceive information from two modalities and output one or more knowledge point descriptions in phrase form that summarize the core semantics of the segment. After extracting knowledge points from all segments, these knowledge point descriptions are used as index keys, and the time interval corresponding to each knowledge point is used as index values, for example: ("function definition syntax", [00:32:15, 00:34:22]), to build an inverted index library. This index library supports matching retrieval based on semantic similarity, rather than relying solely on literal keyword matching. In one specific embodiment, the first pre-trained large model can employ a multimodal pre-trained model based on the Transformer architecture. This model includes a visual encoder (e.g., an encoding structure based on ViT or CLIP visual branches) for receiving and encoding semantic feature vectors from video frames, and a text decoder for receiving the fused features of the semantic feature vectors and the speech-text sequence within the corresponding time window, and autoregressively generating knowledge point descriptions representing the core semantics of the current segment. Through pre-training on a large-scale video-text alignment dataset, this model learns cross-modal semantic understanding and text generation capabilities.

[0019] When a query is received from a user in natural language, the system first converts the query into a semantic feature vector using a semantic encoding model. Then, based on this semantic feature vector, semantic similarity calculation is performed in the pre-built inverted index. This involves calculating the semantic distance between the query semantic feature vector and the knowledge point descriptions corresponding to each index key, and recalling one or more knowledge point descriptions that best match the query semantic feature vector, along with their corresponding candidate time intervals. Since user queries may involve multiple sub-concepts or semantic dimensions, the retrieved results typically contain multiple scattered time intervals. To address this, semantic fusion and redundancy removal operations can be performed: multiple candidate time intervals that semantically belong to the same knowledge point but are temporally scattered or partially overlapping are merged and deduplicated, ultimately integrating them into one or more continuous target time intervals.

[0020] After determining the target time interval, the corresponding video segments are extracted from the original video data based on this time interval as the final output. This extraction operation can be either physical video file splitting or video stream positioning based on timestamps; the specific method is not limited. At this point, the user's query, expressed in natural language, is transformed into a precise target video location interval.

[0021] By applying the technical solution of this embodiment, the system can simultaneously perceive both visual and audio content by extracting semantic feature vectors and speech-text sequences to generate multimodal time-series sequences. This overcomes the limitation of pure ASR caption retrieval, which cannot cover purely visual knowledge points such as charts and demonstration actions. By outputting knowledge point description information through the first pre-trained large model and constructing an inverted index library, semantic-level matching is achieved, overcoming the limitation of traditional keyword retrieval, which can only perform literal matching and cannot understand synonyms, near-synonyms, and contextual semantics. By converting the user's natural language query content into query semantic features and performing semantic relevance calculation, the limitation of keyframe analysis, which can only identify preset categories of objects or scenes and cannot answer complex and abstract questions, is overcome. By performing semantic fusion and redundancy removal operations on candidate time intervals, scattered related knowledge point fragments are integrated into complete target video positioning intervals, reducing the time spent by users searching for specific knowledge points in long videos.

[0022] In this embodiment of the application, optionally, step 103, "converting the natural language query content into corresponding query semantic features, performing semantic relevance calculation in the inverted index based on the query semantic features, and recalling at least one knowledge point description information matching the natural language query content and its corresponding candidate time interval," includes: inputting the natural language query content into a preset intent understanding model, performing semantic parsing on the natural language query content through the intent understanding model, identifying the explicit semantic information and implicit intent information contained therein; the implicit intent information includes the user's expected knowledge granularity level and / or the user's demand type for knowledge content; generating a first query semantic feature based on the explicit semantic information, and... A second query semantic feature is generated based on the implicit intent information; semantic similarity matching is performed in the inverted index based on the first and second query semantic features to obtain at least one candidate knowledge point description information and its corresponding candidate time interval that matches the natural language query content; the user's historical learning behavior data and / or user profile data are obtained to extract the user's personalized preference features; each candidate knowledge point description information is weighted and scored according to the personalized preference features, and each candidate time interval is sorted in descending order according to the weighted scoring results to generate a personalized candidate list containing at least one candidate time interval and its corresponding candidate knowledge point description information, and the personalized candidate list is output as the recall result.

[0023] In this embodiment, the user's natural language query is first fed into a pre-defined intent understanding model. This model performs deep semantic analysis of the natural language query, extracting two levels of information: first, explicit semantic information clearly expressed in the text, such as which concept or entity the user specifically queried; and second, implicit intent information contained in the expression, such as whether the user desires a general explanation or in-depth analysis, or whether the user's query leans towards definitional or operational knowledge. Essentially, this step transforms the user's natural language query from a single string into structured intent data. In a specific embodiment, the intent understanding model can be built based on a pre-trained language model, such as BERT or RoBERTa. In practice, this model achieves semantic analysis of the natural language query by jointly modeling two subtasks: intent recognition and slot filling. The intent recognition task categorizes the user-input query text as a whole, determining which of the predefined intent categories it belongs to, such as "concept explanation," "operation guidance," or "comparative analysis." This classification result represents the user's implicit intent information. The slot filling task assigns corresponding slot labels to each word in the input text, extracting key entity or parameter information contained in the query text, such as technical terminology names, operational objects, or limiting conditions. The extracted information is used to construct the user's explicit semantic information. The two subtasks share the underlying encoding parameters of the intent understanding model. Through joint training, the model simultaneously outputs intent classification results and slot labeling results in a single forward propagation, thereby achieving the synchronous parsing and extraction of explicit semantic information and implicit intent information.

[0024] Subsequently, independent semantic feature vectors are generated for the two types of information mentioned above. The first query semantic feature corresponds to the explicit topic content expressed by the user, answering the question "What is the user asking?"; the second query semantic feature corresponds to the user's implicit request type and knowledge granularity preference, answering the question "How does the user want to obtain the answer?". These two feature vectors characterize the user's query behavior from both content and form dimensions, providing a foundation for subsequent personalized retrieval.

[0025] After obtaining the feature vectors in both dimensions, semantic similarity matching is performed in the inverted index based on both. That is, the retrieval process no longer relies solely on the semantic relevance between the query content and the knowledge point description information; it also considers the user's intent to guide the matching results. This ensures that the retrieved knowledge point description information is not only relevant to the query in content but also more closely matches the user's current expectations in terms of presentation and knowledge depth. Based on a comprehensive consideration of these two features, one or more knowledge point descriptions and their corresponding candidate time intervals that best match the current query are selected.

[0026] Building upon this, further data on the user's historical learning behavior and / or user profile is obtained. Historical learning behavior data includes, for example, previously viewed video clips and knowledge points with longer viewing times; user profile data includes, for example, the user's knowledge background or learning preferences. From this data, preference features that characterize the user's personalized learning habits are extracted. This step allows the system to overlay personalized understandings on top of general semantic retrieval results.

[0027] Finally, the extracted personalized preference features are used to perform a weighted scoring process on the descriptions of each candidate knowledge point retrieved. Knowledge point descriptions that better match user preferences receive higher scores, and their corresponding video time intervals are ranked higher in the final personalized candidate list. The final personalized candidate list is a list containing multiple candidate time intervals and their corresponding knowledge point descriptions, allowing users to select the one that best suits their needs as the final location result.

[0028] This application's embodiments introduce an intent recognition and personalized preference fusion mechanism, enabling the system to dynamically adjust retrieval and ranking strategies based on different expressions of the same query content and different users' learning habits. This upgrades semantic retrieval to personalized knowledge positioning, significantly improving the relevance of retrieval results and the user's learning experience.

[0029] Optionally, in this embodiment of the application, the step of "performing semantic similarity matching in the inverted index database based on the first query semantic features and the second query semantic features to obtain at least one candidate knowledge point description information and its corresponding candidate time interval that matches the natural language query content" includes: fusing the first query semantic features and the second query semantic features to generate a fused query feature; and performing semantic similarity matching in the inverted index database using the fused query feature to obtain at least one candidate knowledge point description information and its corresponding candidate time interval that matches the natural language query content.

[0030] In this embodiment, the first and second query semantic features are first fused to generate a fused query feature. This feature fusion can be achieved using various mathematical operations, such as weighted concatenation of the two feature vectors, element-wise addition, or averaging. The fused feature simultaneously carries information from two dimensions: "what the user is asking" and "how the user hopes to obtain the answer," allowing the subsequent retrieval process to be guided by both the query content and the user's intent.

[0031] Subsequently, based on the fused query feature, semantic similarity matching is performed in the inverted index. Each knowledge point description in the inverted index has been pre-converted into a vector representation. By calculating the similarity (e.g., cosine similarity) between the fused query feature and each knowledge point description, one or more candidate knowledge point descriptions and their corresponding candidate time intervals that best match the fused query feature are selected. Because the fused query feature integrates intent-dimensional information, the search results tend to rank knowledge fragments that align with the user's intent in terms of relevance.

[0032] Compared to using the first query semantic features alone for retrieval, the embodiments of this application achieve joint optimization of content relevance and intent adaptability through feature fusion, so that intent information plays a role in the retrieval stage, rather than just being used as a filtering condition after retrieval, thereby realizing intent perception in the retrieval process itself and improving the accuracy of recall.

[0033] Optionally, in this embodiment, the step of "performing semantic similarity matching in the inverted index based on the first query semantic features and the second query semantic features to obtain at least one candidate knowledge point description information and its corresponding candidate time interval that matches the natural language query content" includes: performing semantic similarity matching in the inverted index based on the first query semantic features to obtain at least one initial candidate knowledge point description information and its corresponding initial candidate time interval that matches the explicit semantic information; obtaining the knowledge granularity level expected by the user and / or the user's demand type for knowledge content contained in the implicit intent information; invoking a preset retrieval strategy that matches the knowledge granularity level and / or the demand type; performing secondary filtering or reordering of the at least one initial candidate knowledge point description information based on the preset retrieval strategy to determine at least one candidate knowledge point description information and its corresponding candidate time interval that matches the natural language query content.

[0034] In this embodiment, firstly, semantic similarity matching is performed in the inverted index based on the first query semantic features. This stage only utilizes explicit semantic information extracted from the user's query content, i.e., which concept or entity the user specifically queried, without involving any intent-level guidance. By calculating the similarity between the first query semantic features and the description information of each knowledge point, a batch of knowledge points and their corresponding time intervals that are semantically most relevant to the query content itself are recalled as initial candidate knowledge point description information and initial candidate time intervals. This stage ensures sufficient coverage of the recall results at the content level.

[0035] Subsequently, based on the implicit intent information represented by the semantic features of the second query—that is, the user's desired level of knowledge granularity and / or type of request—a preset retrieval strategy matching the current intent is invoked from multiple preset retrieval strategies. These preset retrieval strategies can have different filtering criteria or ranking logics. For example, when the user's intent is "general understanding," a strategy that tends to recall more general and coarser-grained knowledge points can be invoked; when the user's intent is "detailed learning," a strategy that tends to retain more specific and in-depth knowledge points is invoked. The initial candidate knowledge point description information is then further filtered or reordered according to the invoked preset retrieval strategy, ensuring that the final output candidate knowledge point description information and its candidate time intervals are not only relevant to the query in content but also consistent with the user's intent preferences in terms of knowledge depth and presentation. In one specific embodiment, the preset retrieval strategy includes retrieval parameter configurations corresponding to different knowledge granularity levels and / or request types. Secondary filtering or re-ranking can be specifically implemented as follows: filtering initial candidate knowledge point description information based on the similarity threshold and / or recall limit configured in the preset retrieval strategy; and / or correcting the similarity score of each initial candidate knowledge point description information based on the knowledge granularity weight factor configured in the preset retrieval strategy. For example, when the implicit intent information represents a coarse-grained knowledge granularity level desired by the user, the granularity weight factor of the corresponding general knowledge point in the initial candidate knowledge point description information is set to a positive number greater than 1 to improve its original similarity score; when the implicit intent information represents a fine-grained knowledge granularity level desired by the user, the granularity weight factor of the corresponding detailed knowledge point in the initial candidate knowledge point description information is set to a positive number greater than 1 to improve its original similarity score. The initial candidate knowledge point description information is then re-ranked based on the corrected similarity score, so that knowledge points matching the user's desired knowledge granularity level receive a higher ranking position.

[0036] In this embodiment, when the search results do not meet expectations, it is possible to clearly pinpoint whether the problem lies in the initial recall phase or the strategy intervention phase, facilitating step-by-step debugging and optimization. Simultaneously, the configurability of the strategy library provides convenience for system operation and maintenance. Administrators can dynamically adjust or expand the preset search strategies corresponding to different intents based on actual application scenarios without modifying the search model itself.

[0037] In this embodiment of the application, optionally, after step 103, the method further includes: extracting target multimodal information corresponding to the target video positioning interval; the target multimodal information includes a continuous speech-text sequence and a continuous semantic feature vector within the target video positioning interval; recombining the target multimodal information according to a preset structured prompt word template to generate enhanced prompt words containing the target multimodal information and structured extraction instructions; inputting the enhanced prompt words into a second pre-trained large model, and outputting a structured knowledge summary corresponding to the natural language query content through the second pre-trained large model in a preset fixed data format; the structured knowledge summary includes a core concept summary, hierarchical details, and related knowledge point fields of the knowledge points corresponding to the target video positioning interval; and integrating the structured knowledge summary with the target video positioning interval. The relevant knowledge points are stored in a preset knowledge base. The associated knowledge point fields in the structured knowledge summary are obtained. Using these fields as query conditions, a semantic relevance search is performed in the inverted index database to retrieve at least one associated video positioning interval related to the knowledge point corresponding to the target video positioning interval. Based on the associated video positioning interval, the corresponding associated structured knowledge summary is obtained from the preset knowledge base. The associated structured knowledge summary is pre-associated with the associated video positioning interval and stored in the preset knowledge base. The associated video positioning interval and the core concept summary in the associated structured knowledge summary are output as proactively recommended content to the user interface. In response to the user's selection of the associated video positioning interval, the hierarchical details in the associated structured knowledge summary are output to the user interface.

[0038] In this embodiment, firstly, richer multimodal information about the target is extracted from the target video localization interval. Specifically, this may include a continuous audio-text sequence within the interval and a continuous semantic feature vector representing the visual content. The former provides complete audio narration content, while the latter carries the visual semantic information within the interval in vector form. Together, they constitute the raw material for subsequent knowledge extraction by the large model.

[0039] Subsequently, this target multimodal information is reorganized according to a pre-defined structured cue word template to generate enhanced cue words. This template can be a carefully designed set of instructions, which not only contains the target multimodal information to be processed but can also embed constraints on the output format and knowledge extraction granularity, such as requiring the model to organize the output content according to a specific structure. Through this reorganization process, the originally fragmented speech text sequences and semantic feature vectors are integrated into a well-structured and context-rich input text, enabling the large model to perform subsequent knowledge extraction tasks based on a full understanding of the fragment content.

[0040] Next, the enhanced prompt is input into the second pre-trained large model, driving it to output a structured knowledge summary in a preset fixed data format. This structured knowledge summary is not a simple text summary, but rather semi-structured data with a fixed field structure generated according to system requirements. Specifically, it can include three fields: a core concept summary to summarize the core knowledge points of the segment in one sentence; hierarchical details to list key details; and related knowledge points to identify other concepts or topics related to this knowledge point. The fixed output format ensures that the knowledge summary has good structure, facilitating subsequent storage, retrieval, and display. In a specific embodiment, the second pre-trained large model can adopt a generative large language model based on the Transformer architecture, such as the GPT series or Wenxin Yiyan. The specific implementation of its output structured knowledge summary can be as follows: by designing prompt word templates that include format constraint instructions and field definition instructions, the prompt words explicitly require the model to generate content according to the specified field structure (such as core concept summary, hierarchical details, and related knowledge point fields) and output format (such as JSON or Markdown). The model follows the instruction constraints during the autoregressive generation process and outputs semi-structured text that conforms to the preset field structure.

[0041] After generating a structured knowledge summary, it is associated with the target video location range and stored in a pre-defined knowledge base. This knowledge base is used to persistently store all video segments processed by the system and their corresponding structured knowledge summaries. This allows any user to directly retrieve the existing structured knowledge summary from the knowledge base when they subsequently locate the same video range again, thereby significantly reducing computational resource consumption and response latency while ensuring functional integrity.

[0042] Building upon this, relevant knowledge point fields can be extracted from the generated structured knowledge summary. The knowledge point content contained in these fields can then be used as new query conditions to perform semantic relevance retrieval again within the constructed inverted index. Essentially, this step achieves cross-video association at the knowledge point level. It doesn't extend the user's current query to other locations within the same video, but rather expands the search scope to the entire video library, recalling relevant segments from other videos that are semantically related to the current knowledge point, along with their pre-stored structured knowledge summaries. The relevant structured knowledge summary here is the summary result generated and stored in the knowledge base when the video segment is first processed. Through the knowledge base's persistent storage mechanism, it can be directly retrieved and used during recommendation without needing to be regenerated.

[0043] Subsequently, the recalled associated video location ranges and their corresponding core concept summaries from the associated structured knowledge summaries are proactively recommended to the user. In this recommendation scenario, only core concept summaries are displayed, rather than complete structured knowledge summaries. The aim is to allow users to quickly understand the core theme of the recommended content to help them determine whether it is worthwhile to learn further, while avoiding information overload from complete notes. When a user becomes interested in a recommended piece of content based on this core concept summary and makes a selection, the pre-stored complete hierarchical details are then displayed to the user, thus completing a progressive information presentation.

[0044] This application embodiment uses a second pre-trained large model to transform unstructured audiovisual content in video clips into structured, reusable knowledge summaries, transforming videos from one-time viewing objects into knowledge that can be persistently stored and retrieved; the introduction of a knowledge base optimizes computational costs, requiring only one processing session for the same clip; and cross-video retrieval driven by associated knowledge point fields achieves a leap from "user-initiated search" to "system-initiated recommendation," enabling users to engage in a systematic rather than fragmented learning experience centered around a specific knowledge point.

[0045] In this embodiment of the application, optionally, after "outputting the hierarchical details in the associated structured knowledge summary to the user interface", the method further includes: obtaining the target knowledge point description information corresponding to the target video positioning interval and the associated knowledge point description information corresponding to the associated video positioning interval; generating knowledge point association records using the target knowledge point description information and the associated knowledge point description information as nodes and the semantic association relationship between the target knowledge point description information and the associated knowledge point description information as edges; storing the knowledge point association records in a preset graph database to construct or update a video knowledge graph; the video The knowledge graph is used to represent the structured semantic network between knowledge points corresponding to different video positioning intervals. When a new learning event triggered by the user or any other user is detected, the knowledge point description information corresponding to the learning event is used as the query starting point. Graph traversal retrieval is performed in the video knowledge graph to obtain at least one recommended knowledge point description information with semantic relationship to the query starting point, its corresponding recommended video positioning interval, and the associated structured knowledge summary corresponding to the recommended video positioning interval. The core concept summary in the recommended video positioning interval and the associated structured knowledge summary is output as proactive recommendation content to the user interface.

[0046] In this embodiment, firstly, two key pieces of information are obtained: one is the description information of the target knowledge points corresponding to the target video positioning interval, i.e., the core knowledge points of the segment currently being learned by the user; the other is the description information of the related knowledge points corresponding to the associated video positioning interval, i.e., the core knowledge points of the related segments recommended to the user by the system. Using these two pieces of knowledge point description information as nodes and the semantic relationship between them as edges, a knowledge point association record is generated.

[0047] Subsequently, the associated record of this knowledge point is stored in a pre-defined graph database, thereby realizing the construction or incremental update of the video knowledge graph. A graph database is a type of database specifically designed for storing and querying point-edge relationship data, and it is more efficient than traditional relational databases in handling multi-hop relationship queries. During this process, if a knowledge point node or associated edge already exists in the graph, it is updated or strengthened to ensure the continuous evolution and improvement of the knowledge graph. This knowledge graph represents the semantic network between knowledge points corresponding to different video positioning intervals in a structured manner, organizing the knowledge points originally scattered across various videos into an interconnected knowledge network through semantic relationships.

[0048] Building upon this foundation, when a new learning event is detected, triggered by the current user or any other user (e.g., the user inputs a new natural language query or clicks on a recommended segment), the knowledge point description information corresponding to that learning event serves as the starting point for the query. A graph traversal search is then performed within the constructed video knowledge graph. Graph traversal search refers to the process of expanding outwards along related edges from one or more starting nodes in a graph database to obtain neighboring nodes. Through this search operation, one or more recommended knowledge points with direct or indirect semantic relationships to the query starting point, along with their corresponding recommended video location ranges and associated structured knowledge summaries, are obtained.

[0049] Finally, the retrieved recommended video location range and the core concept summary in the corresponding associated structured knowledge summary are output to the user interface as proactively pushed content, thereby realizing the upgrade from "user-initiated retrieval" to "system-initiated push".

[0050] Compared to the aforementioned single-step semantic retrieval based solely on related knowledge point fields in the current structured knowledge summary, this application's embodiment introduces a graph database to persistently store and accumulate the relationships between knowledge points, enabling the system to support deeper knowledge discovery. Specifically, graph traversal retrieval can uncover indirectly related knowledge points, such as other knowledge points related to the target knowledge point across different domains, which is difficult to cover with single-step semantic retrieval. As the number of processed videos increases, the video knowledge graph continuously enriches and evolves, enhancing the system's recommendation capabilities and demonstrating the dual-driven technical effect of data and knowledge. The construction of the video knowledge graph serves all users' subsequent learning events, realizing knowledge reuse and sharing from individual learning experiences to a global knowledge network.

[0051] In this embodiment of the application, optionally, the second pre-trained large model includes a note generation model and a note verification model; the structured knowledge summary is generated in the following manner: the enhanced prompt words are input into the note generation model, and the note generation model outputs an initial structured knowledge summary; the enhanced prompt words and the initial structured knowledge summary are input into the note verification model, and the note verification model performs factual consistency verification and content integrity verification on the initial structured knowledge summary, generating verification feedback information; the factual consistency verification is used to detect whether there is a conflict between the statements in the initial structured knowledge summary and the corresponding content in the enhanced prompt words, and the content integrity verification is used to detect whether the initial structured knowledge summary omits key knowledge points in the enhanced prompt words; the verification feedback information is fed back to the note generation model, and the note generation model corrects the initial structured knowledge summary according to the verification feedback information, generating an updated structured knowledge summary.

[0052] In this embodiment, the second pre-trained large model can consist of two functional modules: a note generation model and a note validation model. The note generation model is responsible for performing the initial generation task of structured knowledge summaries, while the note validation model is responsible for quality assessment. Both models can use the same pre-trained large model architecture, but through different fine-tuning training, they can respectively learn generation and discrimination capabilities. First, the constructed enhanced prompts are input into the note generation model. Based on the target multimodal information and format constraints in the enhanced prompts, the model generates an initial structured knowledge summary, which preliminarily conforms to the preset field structure and format requirements.

[0053] Subsequently, the original enhanced prompts and the generated initial structured knowledge summary are input into the note-checking model. The task of this model is not to generate new content, but rather to assess the quality of the initial structured knowledge summary. Specifically, the model performs two checks: factual consistency check, which detects whether each statement in the initial structured knowledge summary conflicts with or distorts the corresponding original content in the enhanced prompts, preventing the model from introducing "illusory" content during generation; and content completeness check, which detects whether the initial structured knowledge summary omits key knowledge points that should be covered by the enhanced prompts. Both checks quantitatively evaluate the summary quality from two dimensions: "fidelity" and "coverage," and ultimately generate validation feedback information containing specific modification suggestions.

[0054] Next, the validation feedback is fed back to the note-generating model, driving it to revise the initial structured knowledge summary based on the suggested modifications. This step utilizes error location hints and suggested revision directions in the validation feedback to specifically adjust the summary content. For example, statements inconsistent with the original content can be deleted, or missing key knowledge points can be added, thereby generating an updated structured knowledge summary. The above "generation-validation-revision" process can be iterated multiple times as needed until the note validation model passes the evaluation or the preset iteration limit is reached, to obtain a final summary of higher quality.

[0055] In one specific embodiment, the note generation model can employ a pre-trained large language model with a decoder-only architecture (such as the GPT series). This model is built upon the decoder part of a Transformer and includes modules such as multi-head self-attention and feedforward neural networks. It is mainly used to predict and generate structured knowledge summary text word by word in an autoregressive manner based on the input enhancement prompts. The note verification model can employ a pre-trained large language model with an encoder-only architecture (such as BERT). This model is built upon the encoder part of a Transformer and includes multi-head self-attention layers and feedforward neural networks. It is mainly used to jointly encode and extract features from the input "enhanced prompts" and "initial structured knowledge summary," and then output verification feedback information to evaluate the factual consistency and content completeness of the summary through a top-level classification or scoring network.

[0056] Compared to conventional single-model generation methods, this application's embodiment introduces a dual-model adversarial verification mechanism, upgrading the quality control of the summary from "relying solely on the generation model's own capabilities" to "mutual competition and iterative optimization between generation and verification," significantly improving the accuracy and reliability of structured knowledge summaries, and thus providing a higher-quality data foundation for subsequent video knowledge graph construction and cross-video recommendation.

[0057] Furthermore, as Figure 1 In terms of specific implementation, this application provides an apparatus for knowledge localization of videos based on a large model, such as... Figure 2 As shown, the device includes: The sequence generation module is used to acquire the original video data, parse the visual modality and audio modality of the video data respectively, extract the semantic feature vector of each video frame and the speech text sequence containing timestamp information, and align the semantic feature vector and the speech text sequence on the time axis to generate a multimodal time sequence. The index library construction module is used to slide and segment the multimodal time series according to a preset time granularity to obtain multiple segments. Each segment is input into the first pre-trained large model. The first pre-trained large model outputs at least one knowledge point description information to represent the core semantics of the current segment. The inverted index library is constructed using the knowledge point description information as the index key and the corresponding time interval as the index value. The video capture module is used to respond to the natural language query content input by the user, convert the natural language query content into corresponding query semantic features, perform semantic relevance calculation in the inverted index database based on the query semantic features, recall at least one knowledge point description information and its corresponding candidate time interval that match the natural language query content, and perform semantic fusion and redundancy removal operations on all recalled candidate time intervals to obtain the target time interval. Based on the target time interval, the module captures the target video positioning interval that covers the natural language query content from the video data.

[0058] Optionally, the video capture module is used for: The natural language query content is input into a preset intent understanding model, and the intent understanding model performs semantic parsing on the natural language query content to identify the explicit semantic information and implicit intent information contained therein; the implicit intent information includes the user's expected knowledge granularity level and / or the user's demand type for knowledge content; A first query semantic feature is generated based on the explicit semantic information, and a second query semantic feature is generated based on the implicit intent information; Based on the first query semantic features and the second query semantic features, semantic similarity matching is performed in the inverted index to obtain at least one candidate knowledge point description information and its corresponding candidate time interval that matches the natural language query content. Obtain users' historical learning behavior data and / or user profile data to extract users' personalized preference features; The description information of each candidate knowledge point is weighted and scored according to the personalized preference features, and the candidate time intervals are sorted in descending order according to the weighted scoring results to generate a personalized candidate list containing at least one candidate time interval and its corresponding candidate knowledge point description information. The personalized candidate list is then output as the recall result.

[0059] Optionally, the video capture module is further configured to: The first query semantic features and the second query semantic features are fused together to generate fused query features; Using the fusion query features, semantic similarity matching is performed in the inverted index to obtain at least one candidate knowledge point description information and its corresponding candidate time interval that matches the natural language query content.

[0060] Optionally, the video capture module is further configured to: Using the first query semantic features, semantic similarity matching is performed in the inverted index to obtain at least one initial candidate knowledge point description information that matches the explicit semantic information and its corresponding initial candidate time interval. Obtain the knowledge granularity level expected by the user and / or the user's demand type for knowledge content contained in the implicit intent information, invoke a preset retrieval strategy that matches the knowledge granularity level and / or the demand type, perform secondary filtering or reordering of the at least one initial candidate knowledge point description information based on the preset retrieval strategy, and determine at least one candidate knowledge point description information and its corresponding candidate time interval that matches the natural language query content.

[0061] Optionally, the device further includes a recommendation module; the recommendation module is configured to: After extracting the target video location interval covering the natural language query content from the video data, the target multimodal information corresponding to the target video location interval is extracted; the target multimodal information includes a continuous speech-text sequence and a continuous semantic feature vector within the target video location interval; The target multimodal information is reorganized according to a preset structured prompt word template to generate an enhanced prompt word that contains the target multimodal information and a structured extraction instruction; The enhanced prompt words are input into the second pre-trained large model, which then outputs a structured knowledge summary corresponding to the natural language query content in a preset fixed data format. The structured knowledge summary includes the core concept summary, hierarchical details, and related knowledge point fields of the knowledge points corresponding to the target video positioning interval. The structured knowledge summary is associated with the target video positioning range and stored in a preset knowledge base; The associated knowledge point fields in the structured knowledge summary are obtained. Using the associated knowledge point fields as query conditions, semantic relevance retrieval is performed in the inverted index to recall at least one associated video positioning interval that is associated with the knowledge point corresponding to the target video positioning interval. Based on the associated video positioning interval, the corresponding associated structured knowledge summary is obtained from the preset knowledge base. The associated structured knowledge summary is pre-associated with the associated video positioning interval and stored in the preset knowledge base. The associated video location range and the core concept summary in the associated structured knowledge summary are output as proactive recommendation content to the user interface; In response to the user's selection of the associated video location range, the hierarchical details in the associated structured knowledge summary are output to the user interface.

[0062] Optionally, the recommendation module is further configured to: After outputting the hierarchical details in the associated structured knowledge summary to the user interface, the target knowledge point description information corresponding to the target video positioning interval and the associated knowledge point description information corresponding to the associated video positioning interval are obtained. Knowledge point association records are generated with the target knowledge point description information and the associated knowledge point description information as nodes and the semantic association relationship between the target knowledge point description information and the associated knowledge point description information as edges. The knowledge points are associated records and stored in a preset graph database to construct or update a video knowledge graph; the video knowledge graph is used to represent the structured semantic network between knowledge points corresponding to different video positioning intervals. When a new learning event triggered by the user or any other user is detected, the knowledge point description information corresponding to the learning event is used as the query starting point. Graph traversal retrieval is performed in the video knowledge graph to obtain at least one recommended knowledge point description information that has a semantic relationship with the query starting point, its corresponding recommended video positioning interval, and the associated structured knowledge summary corresponding to the recommended video positioning interval. The recommended video location range and the core concept summary in the associated structured knowledge summary are output to the user interface as proactive recommended content.

[0063] Optionally, the second pre-trained large model includes a note generation model and a note validation model; the recommendation module is further used for: The enhanced prompt words are input into the note generation model, and the note generation model outputs an initial structured knowledge summary. The enhanced prompts and the initial structured knowledge summary are input into the note verification model. The note verification model performs factual consistency verification and content integrity verification on the initial structured knowledge summary, generating verification feedback information. The factual consistency verification is used to detect whether there is a conflict between the statements in the initial structured knowledge summary and the corresponding content in the enhanced prompts. The content integrity verification is used to detect whether the initial structured knowledge summary omits key knowledge points in the enhanced prompts. The verification feedback information is fed back to the note generation model, which then corrects the initial structured knowledge summary based on the verification feedback information to generate an updated structured knowledge summary.

[0064] It should be noted that other corresponding descriptions of the functional units involved in the device for knowledge localization of video based on a large model provided in this application embodiment can be found in the following references. Figure 1 The corresponding descriptions in the method will not be repeated here.

[0065] This application also provides a computer device, which may specifically be a personal computer, a server, a network device, etc. Figure 3 As shown, the computer device includes a bus, a processor, memory, and a communication interface, and may also include an input / output interface and a display device. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores location information. The network interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0066] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0067] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0068] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0069] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0070] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0071] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0072] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for knowledge localization in videos based on a large model, characterized in that, include: The original video data is acquired, and the visual and audio modalities of the video data are parsed respectively. The semantic feature vectors of each video frame and the speech text sequence containing timestamp information are extracted. The semantic feature vectors and the speech text sequence are aligned on the time axis to generate a multimodal time series sequence. The multimodal time series is slidably segmented according to a preset time granularity to obtain multiple segments. Each segment is input into a first pre-trained large model. The first pre-trained large model outputs at least one knowledge point description information to represent the core semantics of the current segment. An inverted index library is constructed using the knowledge point description information as the index key and the corresponding time interval as the index value. In response to a user's input natural language query, the natural language query is converted into corresponding query semantic features. Based on the query semantic features, semantic relevance is calculated in the inverted index database. At least one knowledge point description information matching the natural language query and its corresponding candidate time interval are recalled. Semantic fusion and redundancy removal operations are performed on all recalled candidate time intervals to obtain a target time interval. Based on the target time interval, a target video positioning interval covering the natural language query is extracted from the video data.

2. The method according to claim 1, characterized in that, The step of converting the natural language query content into corresponding query semantic features, calculating semantic relevance based on the query semantic features in the inverted index, and recalling at least one knowledge point description information and its corresponding candidate time interval that matches the natural language query content includes: The natural language query content is input into a preset intent understanding model, and the intent understanding model performs semantic parsing on the natural language query content to identify the explicit semantic information and implicit intent information contained therein; the implicit intent information includes the user's expected knowledge granularity level and / or the user's demand type for knowledge content; A first query semantic feature is generated based on the explicit semantic information, and a second query semantic feature is generated based on the implicit intent information; Based on the first query semantic features and the second query semantic features, semantic similarity matching is performed in the inverted index to obtain at least one candidate knowledge point description information and its corresponding candidate time interval that matches the natural language query content. Obtain users' historical learning behavior data and / or user profile data to extract users' personalized preference features; The description information of each candidate knowledge point is weighted and scored according to the personalized preference features, and the candidate time intervals are sorted in descending order according to the weighted scoring results to generate a personalized candidate list containing at least one candidate time interval and its corresponding candidate knowledge point description information. The personalized candidate list is then output as the recall result.

3. The method according to claim 2, characterized in that, The step of performing semantic similarity matching in the inverted index based on the first query semantic features and the second query semantic features to obtain at least one candidate knowledge point description information and its corresponding candidate time interval that matches the natural language query content includes: The first query semantic features and the second query semantic features are fused together to generate fused query features; Using the fusion query features, semantic similarity matching is performed in the inverted index to obtain at least one candidate knowledge point description information and its corresponding candidate time interval that matches the natural language query content.

4. The method according to claim 2, characterized in that, The step of performing semantic similarity matching in the inverted index based on the first query semantic features and the second query semantic features to obtain at least one candidate knowledge point description information and its corresponding candidate time interval that matches the natural language query content includes: Using the first query semantic features, semantic similarity matching is performed in the inverted index to obtain at least one initial candidate knowledge point description information that matches the explicit semantic information and its corresponding initial candidate time interval. Obtain the knowledge granularity level expected by the user and / or the user's demand type for knowledge content contained in the implicit intent information, invoke a preset retrieval strategy that matches the knowledge granularity level and / or the demand type, perform secondary filtering or reordering of the at least one initial candidate knowledge point description information based on the preset retrieval strategy, and determine at least one candidate knowledge point description information and its corresponding candidate time interval that matches the natural language query content.

5. The method according to claim 1, characterized in that, After extracting the target video location range covering the natural language query content from the video data, the method further includes: Extract the target multimodal information corresponding to the target video localization interval; the target multimodal information includes a continuous speech-text sequence and a continuous semantic feature vector within the target video localization interval; The target multimodal information is reorganized according to a preset structured prompt word template to generate an enhanced prompt word that contains the target multimodal information and a structured extraction instruction; The enhanced prompt words are input into the second pre-trained large model, which then outputs a structured knowledge summary corresponding to the natural language query content in a preset fixed data format. The structured knowledge summary includes the core concept summary, hierarchical details, and related knowledge point fields of the knowledge points corresponding to the target video positioning interval. The structured knowledge summary is associated with the target video positioning range and stored in a preset knowledge base; The associated knowledge point fields in the structured knowledge summary are obtained. Using the associated knowledge point fields as query conditions, semantic relevance retrieval is performed in the inverted index to recall at least one associated video positioning interval that is associated with the knowledge point corresponding to the target video positioning interval. Based on the associated video positioning interval, the corresponding associated structured knowledge summary is obtained from the preset knowledge base. The associated structured knowledge summary is pre-associated with the associated video positioning interval and stored in the preset knowledge base. The associated video location range and the core concept summary in the associated structured knowledge summary are output as proactive recommendation content to the user interface; In response to the user's selection of the associated video location range, the hierarchical details in the associated structured knowledge summary are output to the user interface.

6. The method according to claim 5, characterized in that, After outputting the hierarchical details from the associated structured knowledge summary to the user interface, the method further includes: Obtain the target knowledge point description information corresponding to the target video positioning interval and the associated knowledge point description information corresponding to the associated video positioning interval. Use the target knowledge point description information and the associated knowledge point description information as nodes and the semantic association relationship between the target knowledge point description information and the associated knowledge point description information as edges to generate knowledge point association records. The knowledge points are associated records and stored in a preset graph database to construct or update a video knowledge graph; the video knowledge graph is used to represent the structured semantic network between knowledge points corresponding to different video positioning intervals. When a new learning event triggered by the user or any other user is detected, the knowledge point description information corresponding to the learning event is used as the query starting point. Graph traversal retrieval is performed in the video knowledge graph to obtain at least one recommended knowledge point description information that has a semantic relationship with the query starting point, its corresponding recommended video positioning interval, and the associated structured knowledge summary corresponding to the recommended video positioning interval. The recommended video location range and the core concept summary in the associated structured knowledge summary are output to the user interface as proactive recommended content.

7. The method according to claim 5 or 6, characterized in that, The second pre-trained large model includes a note generation model and a note verification model; the structured knowledge summary is generated in the following manner: The enhanced prompt words are input into the note generation model, and the note generation model outputs an initial structured knowledge summary. The enhanced prompts and the initial structured knowledge summary are input into the note verification model. The note verification model performs factual consistency verification and content integrity verification on the initial structured knowledge summary, generating verification feedback information. The factual consistency verification is used to detect whether there is a conflict between the statements in the initial structured knowledge summary and the corresponding content in the enhanced prompts. The content integrity verification is used to detect whether the initial structured knowledge summary omits key knowledge points in the enhanced prompts. The verification feedback information is fed back to the note generation model, which then corrects the initial structured knowledge summary based on the verification feedback information to generate an updated structured knowledge summary.

8. A device for knowledge localization of video based on a large model, characterized in that, include: The sequence generation module is used to acquire the original video data, parse the visual modality and audio modality of the video data respectively, extract the semantic feature vector of each video frame and the speech text sequence containing timestamp information, and align the semantic feature vector and the speech text sequence on the time axis to generate a multimodal time sequence. The index library construction module is used to slide and segment the multimodal time series according to a preset time granularity to obtain multiple segments. Each segment is input into the first pre-trained large model. The first pre-trained large model outputs at least one knowledge point description information to represent the core semantics of the current segment. The inverted index library is constructed using the knowledge point description information as the index key and the corresponding time interval as the index value. The video capture module is used to respond to the natural language query content input by the user, convert the natural language query content into corresponding query semantic features, perform semantic relevance calculation in the inverted index database based on the query semantic features, recall at least one knowledge point description information and its corresponding candidate time interval that match the natural language query content, and perform semantic fusion and redundancy removal operations on all recalled candidate time intervals to obtain the target time interval. Based on the target time interval, the module captures the target video positioning interval that covers the natural language query content from the video data.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

Images