Teaching video reconstruction method and system based on intelligent agent

Abstract

The invention discloses a teaching video reconstruction method and system based on an intelligent agent. The method comprises the following steps: extracting a knowledge content text and time information from a teaching video; constructing a semantic vector library and a graph database; constructing a cue word template according to a mode layer of the knowledge graph; defining a core function module of the intelligent agent system; the core scheduler receives a user question and calls the knowledge base retriever to generate a related knowledge content text; the knowledge base retriever queries the graph database to obtain the corresponding time of the video frame required for answering the question; and the video reconstructor reconstructs and generates a teaching video required by the user by using the video frame time and the original video. Precise semantic understanding and matching of teaching video content and user questions are achieved, the teaching video is reconstructed according to the multi-dimensional requirements of the user for knowledge points, learning targets and learning ability, specific field fine adjustment does not need to be conducted on a large model, the method can adapt to teaching video reconstruction of different subjects and difficulty levels, and the user experience is improved. Wide application and popularization values are realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a teaching video reconstruction method and system based on an intelligent agent, and belongs to the application field of intelligent teaching systems. Background Art

[0002] With the rapid development of artificial intelligence and big data technology, the field of education is undergoing a digital and intelligent transformation. Among them, the application of intelligent teaching systems is becoming more and more extensive, especially in personalized learning and precision teaching. However, existing intelligent teaching systems still face many challenges in processing and utilizing teaching video resources.

[0003] Traditional teaching video resource management and video retrieval methods mainly rely on keyword matching and metadata annotation, which makes it difficult to accurately capture the semantic information of video content, resulting in inaccurate retrieval results and unable to meet users' query requirements on learning goals or ability development. When retrieving and providing videos to users, existing methods are difficult to accurately locate specific video clips related to teaching knowledge points, learning goals, and learning abilities that users are concerned about, resulting in low learning efficiency. How to effectively combine large language models with teaching video resources to provide learners with personalized video content in multiple dimensions such as knowledge points, learning goals, and ability development is still an urgent problem to be solved.

[0004] Patent CN202410135130.X aims to solve the problem of automatically generating videos using documents and input texts. It uses a large language model to extract text, tables and pictures from documents, and generates structured text, visual content and pictures to splice into video output; Patent CN202410405951.0 aims to solve the problem of automatic segmentation and editing of videos. It uses videos to generate video content representation, combines plot description text, obtains video segmentation information with the help of a large model, and divides the video into multiple clips according to the segmentation information; Patent CN202410421351.3 aims to solve the problem of labeling knowledge points in classroom audio and video. It uses a large model to perform text transcription and semantic analysis on live classroom audio and video, automatically extracts chapters and knowledge points and performs time marking. It does not involve the extraction, analysis and information expression of learning objectives and learning abilities, nor does it involve the reconstruction of teaching video content; Patent CN202410893508.2 aims to solve the problem of video retrieval model training. It proposes a method for video frame sampling and text annotation using a multimodal large model to improve the training efficiency of cross-modal video retrieval models. These methods do not address the processing of abstract multi-dimensional information such as knowledge points, learning objectives, and learning abilities in integrated teaching videos, nor do they solve the problem of on-demand reconstruction of teaching videos. Summary of the invention

[0005] The present invention discloses a teaching video reconstruction method and system based on an intelligent agent. By extracting knowledge points and related learning objectives, learning abilities, video frames, time points and other information from teaching videos, a vector database and a multidimensional space knowledge graph are constructed, and the semantic understanding of user questions and the accuracy of information retrieval are improved by using these information with the help of a large model, so as to realize on-demand reconstruction of teaching videos. The method comprises: 1) extracting knowledge content text, video frame ID and time information from teaching videos to generate a corresponding relationship table; 2) constructing a semantic vector library and a graph database by using the knowledge content text; 3) constructing a prompt word template according to the pattern layer of the knowledge graph; 4) defining the core functional modules of the intelligent agent system; 5) a core scheduler receives user questions and calls a knowledge base retriever to generate knowledge content text R1 related to the questions; 6) the knowledge base retriever queries the graph database with the user questions and R1 to obtain the knowledge points, learning contents, learning objectives and video frame times related to the learning abilities corresponding to the questions; 7) a video reconstructor reconstructs and generates the teaching video required by the user by using the video frame time, the video frames and voice clips in the original video; specifically, the method of the present invention comprises the following steps:

[0006] A. Extract the knowledge content text, video frame ID and time information from the teaching video and generate the corresponding relationship table E. The specific steps are as follows:

[0007] A1. Use OCR tools to extract knowledge content text from teaching video frame images;

[0008] A2. Use speech recognition tools to extract knowledge content text from teaching video speech;

[0009] A3. Remove duplicates from the knowledge content texts extracted from A1 and A2 and merge them into a time-aligned knowledge content text set T;

[0010] A4. Construct text-time correspondence table E={qt i}, i∈(1,N); qt i =(t i ,s i ,e i ,vfs i ,vfe i );qt i represents the i-th text-time quintuple; N represents the number of knowledge content texts in T, t i represents the i-th knowledge content text, s i and e i Indicates t i The starting and ending time points of the corresponding video and audio, vfs i and vfe i Indicates t iThe corresponding starting video frame ID and ending video frame ID;

[0011] B. Use the knowledge content text to build a semantic vector library and graph database. The specific operations are as follows:

[0012] B1. Use the knowledge content text to build a semantic vector library. The specific operations are as follows:

[0013] B1.1 Use the Embedding model to embed each t i (i∈(1,N)) knowledge content text is converted into a semantic vector;

[0014] B1.2 Store the semantic vector into the vector database and construct the vector database index;

[0015] B2. Use the knowledge content text, video frame ID and time information in E to build a knowledge graph and store it in the graph database. The specific steps are as follows:

[0016] B2.1 Define the knowledge graph model layer. The specific steps are as follows:

[0017] B2.1.1 Define entity category set C = {knowledge point, learning content, learning goal, learning ability};

[0018] B2.1.2 Define a set of relationship categories R, which includes the relationship between knowledge points, the relationship between knowledge points and learning content, the relationship between knowledge points and learning ability, and the relationship between learning content and learning objectives;

[0019] B2.2 Based on the knowledge graph model layer, the knowledge graph data layer is constructed using the knowledge content text, video frame ID and time information in the correspondence table E;

[0020] B2.3 Store the constructed knowledge graph in the graph database;

[0021] C. Construct a prompt word template set based on the pattern layer of the knowledge graph. The specific steps are as follows:

[0022] C1. Construct a prompt word template that allows the large model to generate similar questions related to the user's question, denoted as p1. The specific steps are as follows:

[0023] C1.1 Describe the knowledge graph model layer, including entity categories and relationship categories;

[0024] C1.2 Define the user question variable ${Question}, the value of Question is the user question;

[0025] C1.3 Define the variable ${m}, where the value of m describes the number of similar questions to be generated;

[0026] C2. Construct a few-sample thinking chain prompt word template that allows the large model to generate graph query statements, denoted as p2. The specific steps are as follows:

[0027] C2.1 describes the knowledge graph model layer, including entity categories and relationship categories;

[0028] C2.2 Define user question variable ${Question};

[0029] C2.3 Define the knowledge content text variable ${R1} related to the user's question;

[0030] C2.4 Define the thought chain variable for generating graph database query statements as ${CoT};

[0031] C2.5 Define the few-sample variable for generating graph database query statements as ${Exp};

[0032] D. Define the core functional modules of the intelligent agent system. The specific steps are as follows:

[0033] D1. Define the core scheduler, implement the core process of the agent, call each functional module, pass parameters and data, and ensure that each module works together;

[0034] D2. Define a knowledge base retriever to retrieve the semantic vector library and graph database based on the input user question, and obtain the video frame time related to the knowledge point text, learning objectives, and learning ability that are similar to the question;

[0035] D3. Define a video reconstructor to reconstruct and generate the teaching video required by the user according to the video frame time;

[0036] D4. Define external functions and tool lists. The specific steps are as follows:

[0037] D4.1 Define function F1 to execute graph database query. The specific steps are as follows:

[0038] D4.1.1 Define function parameters and receive graph database query statements as input;

[0039] D4.1.2 Set the connection parameters with the graph database;

[0040] D4.1.2 Execute graph database query statements and return video frame times of knowledge points, learning content, learning goals, and learning capabilities related to user questions;

[0041] D4.2 Define the function F2 for extracting teaching video clips. The specific steps are as follows:

[0042] D4.2.1 Define function parameters and receive video frame time as input;

[0043] D4.2.2 Use FFmpeg tool to extract video and audio clips of specified video frame time period;

[0044] D4.2.3 Extract and merge multiple video and audio clips;

[0045] D4.2.4 Return the extracted merged video segments;

[0046] D4.3 Define the tool list. The specific steps are as follows:

[0047] D4.3.1 Create a tool list;

[0048] D4.3.2 Add tool definitions for F1 and F2, including function name, description, and parameter structure;

[0049] E. The core scheduler receives the user's question and calls the knowledge base retriever to generate the knowledge content text R1 related to the question. The specific steps are as follows:

[0050] E1. The core scheduler generates the prompt word P1 using the user question. The specific steps are as follows:

[0051] E1.1 Substitute the user question into the user question variable ${Question} in p1;

[0052] E1.2 Substitute the number of similar questions generated that are related to the user's question into the variable ${m};

[0053] E2. The knowledge base retriever submits the prompt word P1 to the large model to generate a set Qs containing m similar questions related to the user's question;

[0054] E3. The knowledge base retriever uses the Embedding model in step B1 to convert the user question and the m questions in the question set Qs into m+1 query vectors Q0, Q1,…, Q m ;

[0055] E4. The knowledge base retriever performs a m , perform similarity search in the vector database to obtain the top-k most similar knowledge content texts;

[0056] E5. The knowledge base retriever merges all the knowledge content texts in step E4, removes duplicates, and obtains a candidate knowledge content text set C;

[0057] E6. The knowledge base retriever uses the ranking model to rank the candidate knowledge content text set C to obtain the knowledge point text set R1 corresponding to the question. The specific steps are as follows:

[0058] E6.1 Calculate the relevance score of each candidate knowledge content text to the original question;

[0059] E6.2 Sort candidate knowledge content texts in descending order according to relevance scores;

[0060] E6.3 Select the first n knowledge content texts after sorting as the final search result R1;

[0061] F. The knowledge base retriever queries the graph database with the user question and R1 to obtain the knowledge points, learning content, learning objectives, and video frame time related to learning ability corresponding to the question. The specific steps are as follows:

[0062] F1. The knowledge base retriever replaces the template variables in the prompt template p2 with the corresponding variable values ​​to obtain the prompt word P2 that allows the large model to generate a graph database query statement. The specific steps are as follows:

[0063] F1.1 Substitute the user question into the user question variable ${Question} in p2;

[0064] F1.2 Substitute the knowledge content text related to the user's question into the variable ${R1};

[0065] F1.3 Substitute the thought chain description for generating the graph database query statement into the variable ${CoT};

[0066] F1.4 Substitute the few samples of the generated graph database query statement into the variable ${Exp};

[0067] F2 knowledge base retriever according to the prompt word P2, call the large model to generate a graph query statement Qu;

[0068] F3. The knowledge base retriever uses the query statement Qu generated in step F2 as an input parameter, calls the F1 function in the tool list, executes the query statement in the graph database, and obtains the video frame time related to the knowledge point, learning content, learning objectives and learning ability corresponding to the question;

[0069] G. The video reconstructor uses the video frame time, the video frames in the original video and the voice clips to reconstruct the teaching video required by the user. The specific steps are as follows:

[0070] G1. The video reconstructor takes the video frame time information as input parameters, calls the F2 function in the tool list, and reconstructs the teaching video required by the user;

[0071] G2. Save the reconstructed video in the specified directory.

[0072] Compared with the prior art, the present invention has the following advantages: it realizes a complete on-demand reconstruction method for teaching videos, and accurately expresses the multi-dimensional information such as knowledge points, learning objectives and learning abilities contained in the teaching videos by constructing a semantic vector library and knowledge graph of the teaching video content text; it constructs the core scheduler, knowledge base retriever and video reconstructor module of the intelligent body system, and realizes the accurate semantic understanding of the teaching video content and user questions with the help of a large model, improves the retrieval accuracy of the teaching video content, and reconstructs the teaching video according to the user's multi-dimensional needs for knowledge points, learning objectives and learning abilities. The present invention does not require fine-tuning of large models, and improves the efficiency of application system development; the designed intelligent body architecture and prompt word template make the system flexible and scalable, and can adapt to the reconstruction of teaching videos of different subjects and difficulty levels, and has a wide range of application and promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 :A teaching video reconstruction method based on intelligent agent and system flow chart

[0074] Figure 2 :A teaching video reconstruction method based on intelligent agent and system module diagram DETAILED DESCRIPTION

[0075] The present invention is described in detail below with reference to the accompanying drawings and specific implementation examples.

[0076] The present invention discloses a teaching video reconstruction method and system based on an intelligent agent, including: 1) extracting knowledge content text, video frame ID and time information from the teaching video to generate a corresponding relationship table; 2) using the knowledge content text to build a semantic vector library and a graph database; 3) building a prompt word template according to the pattern layer of the knowledge graph; 4) defining the core functional module of the intelligent agent system; 5) the core scheduler receives the user's question and calls the knowledge base retriever to generate the knowledge content text R1 related to the question; 6) the knowledge base retriever queries the graph database with the user's question and R1 to obtain the video frame time related to the knowledge point, learning content, learning goal and learning ability corresponding to the question; 7) the video reconstructor uses the video frame time, the video frame and the voice clip in the original video to reconstruct and generate the teaching video required by the user. The following steps are taken as an example of the Python programming language teaching video to clearly and completely describe the technical solution in the present invention.

[0077] 1. Extract the knowledge content text, video frame ID and time information from the teaching video and generate the corresponding relationship table E. The specific steps are as follows:

[0078] 1.1 Use PaddleOCR to extract knowledge content text from teaching video frame images;

[0079] 1.2 Use iFlytek speech recognition tools to extract knowledge content text from teaching video voice;

[0080] 1.3 Remove duplicates from the knowledge content texts extracted from A1 and A2 and merge them into a time-aligned knowledge content text set T;

[0081] 1.4 Construct the text-time correspondence table E, some of the content examples are shown in Table 1;

[0082] Table 1: Example of some contents of E

[0083] Knowledge content text Starting time point End time Starting video frame ID End video frame ID The integer type is consistent with the concept of integers in mathematics. It can be positive or negative, and there is no limit on the value range. It can be positive or negative, and there is no limit on the value range. 14260ms 15550ms 342 373 The integer type is consistent with the concept of integers in mathematics. It can be positive or negative, and there is no limit on the value range. pow(x,y) function: calculate x, as large as you want. Here we introduce a function pow(x,y), which is used to calculate x to the power of y. Since integers are unlimited, using pow(x,y) we can achieve the effect of calculating as large as we want. 16850ms 32390ms 404 777 Integer types are consistent with the concept of integers in mathematics. They can be positive or negative, and there is no limit on the value range. pow(x,y) function: calculate x, as large as you want >>> pow(2,100) 1267650600 22822940 1496703205376 For example, we can calculate pow(2,100), and the computer can easily calculate the result 33680ms 42760ms 808 1025

[0084] 2. Use the knowledge content text to build a semantic vector library and graph database. The specific operations are as follows:

[0085] 2.1 Use knowledge content text to build a semantic vector library. The specific operations are as follows:

[0086] 2.1.1 Use the bge-large-zh-v1.5 model to convert each piece of knowledge content text into a semantic vector;

[0087] 2.1.2 Store the semantic vector in the Chroma vector database and build a vector database index;

[0088] 2.2 Use the knowledge content text, video frame ID and time information in E to build a knowledge graph and store it in the graph database. The specific steps are as follows:

[0089] 2.2.1 Define the knowledge graph model layer. The model layer of the knowledge graph is shown in Table 2;

[0090] Table 2: Schema layers of the knowledge graph

[0091] Header Entity Class Relationship Category Tail Entity Class Knowledge Points Containment relationship Knowledge Points Knowledge Points Dependencies Knowledge Points Knowledge Points Same relationship Knowledge Points Knowledge Points Conceptual Relationship definition Knowledge Points Explain the relationship describe Knowledge Points Method Relationship usage Knowledge Points Example relationship Examples Knowledge Points nourish Learning ability definition Completion learn describe Completion understand method Completion master Examples Completion use

[0092] Among them, the four categories of definition, description, method and example are subcategories of the learning content category;

[0093] 2.2.2 According to the knowledge graph model layer, the knowledge graph data layer is constructed using the knowledge content text, video frame ID and time information in the correspondence table E;

[0094] 2.2.3 Store the constructed knowledge graph in the gStore graph database;

[0095] 3. Construct a prompt word template set based on the pattern layer of the knowledge graph. The specific steps are as follows:

[0096] 3.1 Construct a prompt word template that allows the large model to generate similar questions related to the user's question, denoted as p1. The content of p1 is shown in Table 3;

[0097] Table 3: Prompt word template p1

[0098] Prompt word template p1 You are an assistant who helps users solve Python programming language problems. Your task is to generate ${m} similar questions related to the user's question ${Question}. You need to combine the defined knowledge graph pattern layer, consider the knowledge points, learning content, learning goals and learning ability in the user's question, and finally output similar questions. Rules of Python programming language knowledge graph pattern layer: Entity category: -Knowledge point class: represents specific concepts or topics in Python programming language. -Learning content class: includes definition class, description class, usage class, and instance class. -Learning goal class: includes understanding class, comprehension class, mastery class, and application class. -Learning ability class: includes data processing ability, text processing ability, computational thinking ability, logical programming ability, and ecological application ability. Relationship category: -Knowledge point class and knowledge point class: including relationship, dependency relationship, and same relationship. -Knowledge point class and learning content class: achieve relationship (learning a certain learning content can achieve a certain learning goal). -Conceptual relationship (connecting definition class) -Explanation relationship (connecting description class) -Method relationship (connecting usage class) -Illustration relationship (connecting instance class) Knowledge point class and learning ability class: Cultivation relationship (learning a certain knowledge point can cultivate a certain ability). Learning content class and learning goal class: -Definition class → Understanding class -Description class → Understanding class -Usage class → Mastery class -Instance class → Application class

[0099] 3.2 Construct a few-sample thinking chain prompt word template that allows the large model to generate graph query statements, denoted as p2. The content of p2 is shown in Table 4:

[0100] Table 4: Prompt word template p2

[0101] Prompt word template p2 You are an assistant who specializes in processing gStore knowledge graph queries. You need to generate SPARQL query statements that conform to the Python programming language knowledge graph pattern layer based on the user's natural language questions and related knowledge content R1. Your answer should strictly follow the pattern layer structure of the knowledge graph to ensure the accuracy and effectiveness of the query statement. User question: ${user question} Related knowledge content R1: ${R1} Python programming language knowledge graph pattern layer rules: Entity categories: - Knowledge point class: represents a specific concept or theme in the Python programming language. - Learning content class: includes definition class, description class, usage class, and instance class. - Learning goal class: includes understanding class, comprehension class, mastery class, and application class. - Learning ability class: includes data processing ability, text processing ability, computational thinking ability, logical programming ability, and ecological application ability. Relationship category: - Knowledge point class and knowledge point class: including relationship, dependency relationship, and identity relationship. - Knowledge point class and learning content class: achieve relationship (learning a certain learning content can achieve a certain learning goal). -Conceptual relationship (connecting definition class) -Explanation relationship (connecting description class) -Method relationship (connecting usage class) -Illustration relationship (connecting instance class) Knowledge point class and learning ability class: Cultivation relationship (learning a certain knowledge point can cultivate a certain ability). Learning content class and learning goal class: -Definition class → Understanding class -Description class → Understanding class -Usage class → Mastery class -Instance class → Application class Entity attributes: Start time and end time: Each learning content class entity has a corresponding time attribute, which indicates the time in the teaching video (in milliseconds). Text attribute: The entities of definition class, description class, usage class and instance class have specific text content. Entity format: The representation format of all entities is <category / entity name>, for example: <knowledge point / function> represents the knowledge point entity of function, <definition / function_definition> represents the definition entity of function, and <ability direction / computational thinking ability> represents the ability direction of computational thinking ability. Task requirements: ${CoT} Example question and analysis: ${Exp}

[0102] 4. Define the core functional modules of the intelligent system. The specific steps are as follows:

[0103] 4.1 Define the core scheduler, implement the core process of the agent, call each functional module, pass parameters and data, and ensure that each module works together;

[0104] 4.2 Define a knowledge base retriever to retrieve the semantic vector library and graph database according to the input user question, and obtain the video frame time related to the knowledge points, learning content, learning objectives and learning ability that are similar to the question;

[0105] 4.3 Define the video reconstructor to reconstruct the teaching video required by the user according to the video frame time;

[0106] 4.4 Define external functions and tool lists. The specific steps are as follows:

[0107] 4.4.1 Define the function F1 for executing gStore graph database query as execute_gstore_query. The specific steps are as follows:

[0108] 4.4.1.1 Define function parameters and receive gStore graph database query statements as input;

[0109] 4.4.1.2 Set the connection parameters with the gStore graph database, including gSore's IP, port, user name, password, query library name, query result return format, etc.;

[0110] 4.4.1.3 Execute SPARQL query statements and return the video frame time of knowledge points, learning content, learning objectives and learning abilities related to the question;

[0111] 4.4.2 Define the function F2 for extracting teaching video clips as extract_video_clips. The specific steps are as follows:

[0112] 4.4.2.1 Define function parameters and receive video frame time as input;

[0113] 4.4.2.2 Use FFmpeg tool to extract video and audio clips of the specified video frame time period;

[0114] 4.4.2.3 Extract and merge multiple video and audio clips;

[0115] 4.4.2.4 Return the extracted merged video segments;

[0116] 4.4.3 Define the tool list. The detailed information of the tool list is shown in Table 5.

[0117] Table 5: Details of tool list

[0118] type name describe parameter Parameter Type Required parameters function execute_gstore_query Execute SPARQL queries in the gStore graph database sparql_query String yes function extract_video_clips Extract segments from video based on time period information time_references String yes

[0119] 5. The core scheduler receives the user's question and calls the knowledge base retriever to generate the knowledge content text R1 related to the question. The specific steps are as follows:

[0120] 5.1 The core scheduler uses the user's question to generate the prompt word P1. The specific steps are as follows:

[0121] 5.1.1 Substitute the user question "How are integers represented in computers? I want to know the various ways to represent integers." into the user question variable ${Question} in p1;

[0122] 5.1.2 Substitute the number of similar questions generated that are related to the user's question into the variable ${m};

[0123] The content of the generated prompt word P1 is shown in Table 6;

[0124] Table 6: Prompt word P1

[0125] Prompt word P1 You are an assistant who helps users solve Python programming language problems. Your task is to generate 3 similar questions related to the user's question "How are integers represented in computers? I want to know the representation methods of various integers." You need to combine the defined knowledge graph pattern layer, consider the knowledge points, learning content, learning goals and learning abilities in the user's questions, and finally output similar questions. Rules of Python programming language knowledge graph pattern layer: Entity categories: -Knowledge point class: represents specific concepts or topics in Python programming language. -Learning content class: includes definition class, description class, usage class, and instance class. -Learning goal class: includes understanding class, comprehension class, mastery class, and application class. -Learning ability class: includes data processing ability, text processing ability, computational thinking ability, logical programming ability, and ecological application ability. Relationship category: -Knowledge point class and knowledge point class: including relationship, dependency relationship, and identical relationship. -Knowledge point class and learning content class: achieve relationship (learning a certain learning content can achieve a certain learning goal). -Conceptual relationship (connecting definition class) -Explanation relationship (connecting description class) -Method relationship (connecting usage class) -Illustration relationship (connecting instance class) Knowledge point class and ability direction class: Cultivation relationship (learning a certain knowledge point can cultivate a certain ability). Learning content class and learning goal class: -Definition class → Understanding class -Description class → Understanding class -Usage class → Mastery class -Instance class → Application class

[0126] 5.2 The knowledge base retriever submits the prompt word P1 to the large model and generates a set of three similar questions related to the user's question: Qs = {"What are the base representations of integers?", "What are the binary, octal, decimal and hexadecimal representations of integers?", "What are the definitions of binary, octal, decimal and hexadecimal?"};

[0127] 5.3 The knowledge base retriever uses the Embedding model in step 2.1.1 to convert the user question and the three questions in the question set Qs into four query vectors Q0, Q1,…, Q3;

[0128] 5.4 Knowledge Base Retriever For each query vector Q0, Q1,…, Q3, a similarity search is performed in the vector database to obtain the top-3 most similar knowledge content texts, as shown in Table 7;

[0129] Table 7: Query vectors and corresponding top-3 knowledge content texts

[0130]

[0131] 5.5 Knowledge Base Retriever Merge all knowledge content texts in E4, remove duplicates, and obtain the candidate knowledge content text set C. The content of C is shown in Table 8;

[0132] Table 8: Candidate knowledge content text set C

[0133] Serial number Knowledge content text 1 There are four types of integer representations: decimal: 1010, 99, -217 For example, 1010, 99, -217 are all decimal representations and are the most commonly used. 2 Integer type has 4 bases of representation Decimal: 1010, 99, -217 Binary, starting with 0b or 0B: 0b010, -0B101 Further, it has a binary representation starting with 0b where b can be uppercase or lowercase, such as 0b010-0B101, etc. Then integers can also be represented in octal. 3 Integer type has 4 bases Decimal: 1010, 99, -217 Binary, starts with 0b or 0B: 0b010, -0B101 Octal, starts with 00 or 00: 0o123, -00456 Integers can also be represented in octal, starting with 0o or 0O, such as 0123-00456. Note that 0o is not 00. 4 Integer type has 4 bases Decimal: 1010, 99, -217 Binary, starting with 0b or 0B: 0b010, -0B101 Octal, starting with 00 or 00: 0o123, -00456 Hexadecimal, starting with 0x or 0X: 0x9a, -0X89 Integers can also be represented in hexadecimal with 0z starting with 0x or 0, such as 0x9a-0X89

[0134] 5.6 The knowledge base retriever uses the bge-reranker-large model to sort the candidate knowledge content text set C and obtain the knowledge content text set R1 corresponding to the question. The specific steps are as follows:

[0135] 5.6.1 Calculate the relevance score of each candidate knowledge content text to the original question;

[0136] 5.6.2 Sort candidate knowledge content texts in descending order according to relevance scores;

[0137] 5.6.3 Select the first two knowledge content texts after sorting as the final search results R1={"Integer type has 4 base representations Decimal: 1010, 99, -217 For example, 1010, 99, -217 These are decimal representations and are the most commonly used", "Integer type has 4 base representations Decimal: 1010, 99, -217 Binary, starting with 0b or 0B: 0b010, -0B101 Octal, starting with 00 or 00: 0o123, -00456 Hexadecimal, starting with 0x or 0X: 0x9a, -0X89 Integers can also be represented in hexadecimal with 0z starting with 0x or 0, such as 0x9a-0X89"};

[0138] 6. The knowledge base retriever queries the graph database with the user question and R1 to obtain the video frame time related to the knowledge point, learning content, learning goal and learning ability corresponding to the question. The specific steps are as follows:

[0139] 6.1 The knowledge base retriever replaces the template variables in the prompt template p2 with the corresponding variable values ​​to obtain the prompt word P2 that allows the large model to generate a graph database query statement, as shown in Table 9:

[0140] Table 9: Prompt word P2

[0141] Prompt word P2 You are an assistant who specializes in processing gStore knowledge graph queries. You need to generate SPARQL query statements that conform to the Python programming language knowledge graph model layer based on the user's natural language questions and related knowledge content R1. Your answer should strictly follow the model layer structure of the knowledge graph to ensure the accuracy and effectiveness of the query statement. User question: How are integers represented in computers? I want to know various ways to represent integers. Related knowledge content R1={"Integer types have 4 base representations. Decimal: 1010, 99, -217. For example, 1010, 99, -217 are all decimal representations and are the most commonly used", "Integer types have 4 base representations. Decimal: 1010, 99, -217. Binary, starting with 0b or 0B: 0b010, -0B101. Octal, starting with 00 or 00: 0o123, -00456. Hexadecimal, starting with 0x or 0X: 0x9a, -0X89. Integers can also be represented in hexadecimal with 0z, where 0x or 0 starts, such as 0x9a-0X89"}: Python programming language knowledge graph pattern layer rules: Entity category: -Knowledge point class: represents specific concepts or topics in the Python programming language. -Learning content class: includes definition class, description class, usage class, and instance class. -Learning goal class: including understanding class, comprehension class, mastery class, and application class. -Learning ability class: including data processing ability, text processing ability, computational thinking ability, logical programming ability, and ecological application ability. Relationship category: -Knowledge point class and knowledge point class: including relationship, dependency relationship, and identity relationship. -Knowledge point class and learning content class: achievement relationship (learning a certain learning content can achieve a certain learning goal). -Concept relationship (connecting definition class) -Explanation relationship (connecting description class) -Method relationship (connecting usage class) -Illustration relationship (connecting instance class) Knowledge point class and ability direction class: cultivation relationship (learning a certain knowledge point can cultivate a certain ability). Learning content class and learning goal class: -Definition class → understanding class -Description class → comprehension class -Usage class → mastery class -Instance class → application class Entity attributes: start time and end time: Each learning content class entity has a corresponding time attribute, which indicates the time in the teaching video (in milliseconds). Text attribute: The entities of definition class, description class, usage class, and instance class have specific text content. Entity format: All entities are represented in the form of <category / entity name>, for example: <knowledge point / function> represents the knowledge point entity of function, <definition / function_definition> represents the definition entity of function, and <ability direction / computational thinking ability> represents the ability direction of computational thinking ability.Task requirements: - Based on the user's question and the relevant knowledge content in R1, generate SPARQL query statements according to the following steps: - Parse the user's question: Identify the learning objectives (such as understanding functions, understanding binary) or learning abilities (such as text processing ability, computational thinking ability, etc.) in the user's question, and infer the knowledge points or learning content that the user wants to query. - Combine with R1: Use the context information provided by the relevant knowledge content R1 retrieved from step E to determine potential knowledge points or concepts and enhance the accuracy of the query. - Determine the query logic: Based on the knowledge graph pattern layer, combine the user's question and R1 to generate a SPARQL query statement that meets the user's needs, including the start time and end time of the retrieval learning content. - Combine query logic: If the query involves learning content (such as definition, description, usage, example), retrieve the corresponding start time and end time at the same time. - Output query statement: Return the generated query statement to the user and ensure the accuracy of the statement. Example question and analysis: Example 1: User question: I want to process text files downloaded from the Internet. What content do I need to learn? Related knowledge content R1: Common functions for processing text in Python are read() and write(), which are closely related to text file operations. Analysis process: - Identify the ability direction: The user mentioned "processing text files", and the corresponding ability direction is text processing ability. - Combined with R1 information: R1 provides common functions for text processing in Python, so we can infer that knowledge points related to text processing ability may involve content related to file operations, such as read() and write() functions. - Combined query logic: In addition to retrieving related learning content, it is also necessary to obtain the start time and end time of the learning content corresponding to each knowledge point in the video.-SPARQL query statement: SELECT ?learning_content ?start_time ?end_time WHERE { ?knowledge_point <cultivation> <ability direction / text processing ability> . FILTER(?knowledge_point IN (<knowledge point / read>, <knowledge point / write>)) . { ?knowledge_point<conceptual relationship> ?learning_content .} UNION { ?knowledge_point <explanation relationship> ?learning_content .} UNION { ?knowledge_point <method relationship> ?learning_content .} UNION { ?knowledge_point <example relationship> ?learning_content.} ?learning_content <start time> ?start_time . ?learning_content <end time> ?end_time .} Example 2: User question: I want to understand functions and be able to use functions to solve problems. Related knowledge content R1: Function is a basic programming structure used to perform specific tasks. Common examples include sum function and recursive function. Analysis process: - Identify learning objectives: The user mentioned "understand functions and be able to use functions", which needs to involve the two learning objectives of "understand_function" and "use_function". - Combined with R1 information: R1 mentioned "function is a basic programming structure used to perform specific tasks" and "common examples include sum function and recursive function", we can infer that the knowledge points related to functions include the definition, description and some specific examples of functions, such as sum function and recursive function. - Determine query logic: According to user needs, we need to retrieve the definition and description classes related to "understand_function" and the instance classes related to "use_function". In addition, it is necessary to obtain the start time and end time of each learning content in the video. - Combine query logic: Generate SPARQL query statements to retrieve relevant learning content and time attributes.- SPARQL query statement: SELECT ?definition ?example ?start_time ?end_time WHERE { ?definition <achieve> <understand / understand_function>. ?example <achieve> <apply / apply_function>. FILTER(?example IN (<instance / sum_function>, <instance / recursive_function>)). ?definition <start_time> ?start_time. ?definition <end_time> ?end_time. ?example <start_time> ?start_time. ?example <end_time> ?end_time.}。

[0142] 6.2 The knowledge base retriever calls the large model to generate a SPARQL query statement Qu according to P2. The query statement Qu is shown in Table 10:

[0143] Table 10: Query statement Qu

[0144] Query statement SELECT ?definition ?start_time ?end_time WHERE { <knowledge point / integer type representation> <includes> ?knowledge . ?knowledge <conceptual relationship> ?definition . ?definition <achievement> ?understand . FILTER regex(str(?knowledge), "decimal|binary|octal|hexadecimal", "i") . FILTER CONTAINS(str(?understand), "understand") . ?definition <start time> ?start_time . ?definition <end time> ?end_time .}

[0145] 6.3 The knowledge base retriever uses the SPARQL query statement Qu generated in step 6.2 as an input parameter, calls the execute_gstore_query function in the tool list, executes the query statement in the graph database, and obtains the video frame time related to the knowledge point, learning content, learning goal and learning ability corresponding to the question. The knowledge point, learning content, learning goal and learning ability corresponding to the question and their corresponding time information are shown in Table 11;

[0146] Table 11: Knowledge points corresponding to user questions, learning content and their corresponding time information

[0147] Knowledge Points What you will learn Start time End Time Binary Definition / binary_definition: binary, starts with 0b or 0B 67370ms 73850ms Decimal Definition / decimal_Definition: Python integers provide four base representations, the most commonly used is decimal 57000ms 66070ms Octal Definition / Octal_Definition: Octal, starting with 00 or 00 75150ms 89410ms hexadecimal Definition / Hexadecimal_Definition: Hexadecimal, starting with 0x or 0X 90710ms 104970ms

[0148] 7. The video reconstructor uses the video frame time, the video frames and voice clips in the original video to reconstruct the teaching video required by the user. The specific steps are as follows:

[0149] 7.1 The video reconstructor uses the video frame time information in step 6.3 to call the extract_video_clips function in the tool list to reconstruct the teaching video required by the user;

[0150] 7.2 Save the reconstructed video in the specified directory.

[0151] Compared with the prior art, the present invention has the following advantages: it realizes a complete on-demand reconstruction method for teaching videos, and accurately expresses the multi-dimensional information such as knowledge points, learning objectives and learning abilities contained in the teaching videos by constructing a semantic vector library and knowledge graph of the teaching video content text; it constructs the core scheduler, knowledge base retriever and video reconstructor module of the intelligent body system, and realizes the accurate semantic understanding of the teaching video content and user questions with the help of a large model, improves the retrieval accuracy of the teaching video content, and reconstructs the teaching video according to the user's multi-dimensional needs for knowledge points, learning objectives and learning abilities. The present invention does not require fine-tuning of large models, and improves the efficiency of application system development; the designed intelligent body architecture and prompt word template make the system flexible and scalable, and can adapt to the reconstruction of teaching videos of different subjects and difficulty levels, and has a wide range of application and promotion value.

Claims

1. A teaching video reconstruction method and system based on an intelligent agent, the steps comprising: A. Extract the knowledge content text, video frame ID and time information from the teaching video and generate the corresponding relationship table E. The specific steps are as follows: A1. Use OCR tools to extract knowledge content text from teaching video frame images; A2. Use speech recognition tools to extract knowledge content text from teaching video speech; A3. Remove duplicates from the knowledge content texts extracted from A1 and A2 and merge them into a time-aligned knowledge content text set T; A4. Construct a text-time correspondence table E={qti}, i∈(1,N); qti=(ti,si,ei,vfsi,vfei); qti represents the i-th text-time quintuple; N represents the number of knowledge content texts in T, ti represents the i-th knowledge content text, si and ei represent the starting time point and ending time point of the video and audio corresponding to ti, and vfsi and vfei represent the starting video frame ID and ending video frame ID corresponding to ti; B. Use the knowledge content text to build a semantic vector library and graph database. The specific operations are as follows: B1. Use the knowledge content text to build a semantic vector library. The specific operations are as follows: B1.1 Use the Embedding model to convert each knowledge content text ti (i∈(1,N)) into a semantic vector; B1.2 Store the semantic vector into the vector database and construct the vector database index; B2. Use the knowledge content text, video frame ID and time information in E to build a knowledge graph and store it in the graph database; C. Construct a prompt word template set based on the pattern layer of the knowledge graph. The specific steps are as follows: C1. Construct a prompt word template that allows the large model to generate similar questions related to the user's question, denoted as p1; C2. Construct a few-sample thinking chain prompt word template that allows the large model to generate graph query statements, denoted as p2; D. Define the core functional modules of the intelligent agent system. The specific steps are as follows: D1. Define the core scheduler, implement the core process of the agent, call each functional module, pass parameters and data, and ensure that each module works together; D2. Define a knowledge base retriever to retrieve the semantic vector library and graph database based on the input user question, and obtain the video frame time related to the knowledge point text, learning objectives, and learning ability that are similar to the question; D3. Define a video reconstructor to reconstruct and generate the teaching video required by the user according to the video frame time; D4. Define external functions and tool lists; E. The core scheduler receives the user's question and calls the knowledge base retriever to generate the knowledge content text R1 related to the question. The specific steps are as follows: E1. The core scheduler generates the prompt word P1 using the user's question. The specific steps are as follows: E1.1 Substitute the user question into the user question variable ${Question} in p1; E1.2 Substitute the number of similar questions generated that are related to the user's question into the variable ${m}; E2. The knowledge base retriever submits the prompt word P1 to the large model to generate a set Qs containing m similar questions related to the user's question; E3. The knowledge base retriever uses the Embedding model in step B1 to convert the user question and the m questions in the question set Qs into m+1 query vectors Q0, Q1,…, Qm; E4. The knowledge base retriever performs a similarity search in the vector database for each query vector Q0, Q1,…, Qm, and obtains the top-k most similar knowledge content texts; E5. The knowledge base retriever merges all the knowledge content texts in step E4, removes duplicates, and obtains a candidate knowledge content text set C; E6. The knowledge base retriever uses the ranking model to rank the candidate knowledge content text set C to obtain the knowledge point text set R1 corresponding to the question; F. The knowledge base retriever queries the graph database with the user question and R1 to obtain the knowledge points, learning content, learning goals, and video frame time related to learning ability corresponding to the question. The specific steps are as follows: F1. The knowledge base retriever replaces the template variables in the prompt template p2 with the corresponding variable values ​​to obtain the prompt word P2 that allows the large model to generate a graph database query statement. The specific steps are as follows: F1.1 Substitute the user question into the user question variable ${Question} in p2; F1.2 Substitute the knowledge content text related to the user's question into the variable ${R1}; F1.3 Substitute the thought chain description for generating the graph database query statement into the variable ${CoT}; F1.4 Substitute the few samples of the generated graph database query statement into the variable ${Exp}; F2 knowledge base retriever according to the prompt word P2, call the large model to generate a graph query statement Qu; F3. The knowledge base retriever uses the query statement Qu generated in step F2 as an input parameter, calls the F1 function in the tool list, executes the query statement in the graph database, and obtains the video frame time related to the knowledge point, learning content, learning objectives and learning ability corresponding to the question; G. The video reconstructor uses the video frame time, the video frames in the original video and the voice clips to reconstruct the teaching video required by the user. The specific steps are as follows: G1. The video reconstructor takes the video frame time information as input parameters, calls the F2 function in the tool list, and reconstructs the teaching video required by the user; G2. Save the reconstructed video in the specified directory.

2. The agent-based teaching video reconstruction method and system as claimed in claim 1, using the knowledge content text, video frame ID and time information in E to construct a knowledge graph and store it in a graph database, the specific steps are as follows: B2.1 Define the knowledge graph model layer. The specific steps are as follows: B2.1.1 Define entity category set C = {knowledge point, learning content, learning goal, learning ability}; B2.1.2 Define a set of relationship categories R, which includes the relationship between knowledge points, the relationship between knowledge points and learning content, the relationship between knowledge points and learning ability, and the relationship between learning content and learning objectives; B2.2 Based on the knowledge graph model layer, the knowledge graph data layer is constructed using the knowledge content text, video frame ID and time information in the correspondence table E; B2.3 Store the constructed knowledge graph in the graph database.

3. The agent-based teaching video reconstruction method and system as claimed in claim 1, constructing a prompt word template for the large model to generate similar questions related to the user's question, denoted as p1, and the specific steps are as follows: C1.1 Describe the knowledge graph model layer, including entity categories and relationship categories; C1.2 Define the user question variable ${Question}, the value of Question is the user question; C1.3 Define the variable ${m}, where the value of m describes the number of similar questions to be generated.

4. The agent-based teaching video reconstruction method and system as claimed in claim 1, constructs a small sample thinking chain prompt word template for allowing the large model to generate a graph query sentence, denoted as p2, and the specific steps are as follows: C2.1 describes the knowledge graph model layer, including entity categories and relationship categories; C2.2 Define user question variable ${Question}; C2.3 Define the knowledge content text variable ${R1} related to the user's question; C2.4 Define the thought chain variable for generating graph database query statements as ${CoT}; C2.5 Define the few-sample variable for generating graph database query statements as ${Exp}.

5. The agent-based teaching video reconstruction method and system as claimed in claim 1, wherein the external function and tool list are defined, and the specific steps are as follows: D4.1 Define function F1 to execute graph database query. The specific steps are as follows: D4.1.1 Define function parameters and receive graph database query statements as input; D4.1.2 Set the connection parameters with the graph database; D4.1.3 Execute graph database query statements and return video frame times of knowledge points, learning content, learning goals, and learning capabilities related to user questions; D4.2 Define the function F2 for extracting teaching video clips. The specific steps are as follows: D4.2.1 Define function parameters and receive video frame time as input; D4.2.2 Use FFmpeg tool to extract video and audio clips of specified video frame time period; D4.2.3 Extract and merge multiple video and audio clips; D4.2.4 Return the extracted merged video segments; D4.3 Define the tool list. The specific steps are as follows: D4.3.1 Create a tool list; D4.3.2 Add tool definitions for F1 and F2, including function name, description, and parameter structure.

6. According to the agent-based teaching video reconstruction method and system as claimed in claim 1, the knowledge base searcher uses the sorting model to sort the candidate knowledge content text set C to obtain the knowledge point text set R1 corresponding to the question, and the specific steps are as follows: E6.1 Calculate the relevance score of each candidate knowledge content text to the original question; E6.2 Sort candidate knowledge content texts in descending order according to relevance scores; E6.3 Select the first n knowledge content texts after sorting as the final retrieval result R1.

Images