A method and system for automatically associating classroom recording videos with knowledge graphs

Abstract

The present invention provides a method and system for automatically associating classroom recording videos with knowledge graphs, and relates to the field of data processing technology. The present invention utilizes the rule that under normal circumstances, there must be continuous segments in a teaching video that belong to the same knowledge point or continuous knowledge points, adopts a derivation calculation starting from a point to perform forward and backward knowledge point matching, and realizes calculation from point to surface, thereby realizing the association between the entire video and the knowledge points, and the sequential relationship of the teaching knowledge conforms to the objective distribution law. In the derivation process, the knowledge point continuity rule is combined with a large model. When it is not necessary, the rule is used for association processing, and the large model is used for auxiliary judgment only when it is necessary. The setting of fixed rules reduces the amount of calculation, can achieve a great improvement in efficiency, and maintains the objectivity of the matching results and avoids the influence of hallucinations to the greatest extent.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for automatically associating classroom recording videos with knowledge graphs. Background Art

[0002] In current teaching scenarios, knowledge graphs, as the core data of teaching, have a large number of application scenarios in smart education. By automatically associating knowledge nodes in knowledge graphs with classroom videos, students can quickly complete targeted knowledge learning and effectively improve learning efficiency.

[0003] Currently, there are two methods: manual association and matching through large models. However, a large number of classroom videos are generated every day, and manual association has high labor costs and poor timeliness. When all large models are used for matching, there will be problems with slow speed, and the large model is affected by hallucinations, and the matching quality is not high enough.

[0004] Therefore, the present invention provides a method and system for automatically associating classroom recording videos with knowledge graphs to solve the technical problems of high usage cost, poor matching quality and low processing efficiency in the existing technology. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a method and system for automatically associating classroom recording videos with knowledge graphs, aiming to solve the technical problems of high usage cost, poor matching quality and low processing efficiency. Compared with the current matching method, the present invention only uses a large model for auxiliary judgment when necessary. The setting of fixed rules reduces the amount of calculation, can achieve a significant improvement in efficiency, and maintains the objectivity of the matching results and avoids the influence of hallucinations to the greatest extent.

[0006] To achieve the above objectives, this application proposes a method for automatically associating classroom recording videos with knowledge graphs, including:

[0007] Step 1: Preprocess the content of the classroom video to obtain a segment array;

[0008] Step 2: Obtain and generate the corresponding knowledge graph array based on the knowledge graph list;

[0009] Step 3: For each fragment element in the fragment array, associate the matching scores with all knowledge graph elements in the knowledge graph array, obtain the fragment score array and add it to the corresponding fragment element;

[0010] Step 4: Calculate the continuity cumulative score of the fragment elements according to the fragment score array and add it to the fragment elements of the fragment array. Use the continuity cumulative score of each fragment element as the score element and generate a continuity score array.

[0011] Step 5: Query the score element with the highest cumulative continuity score in the continuity score array, and set the fragment element corresponding to the score element as the derivation starting point;

[0012] Step 6: Based on the derivation starting point, perform forward derivation on each fragment element in a forward recursive manner to obtain the fragment elements that meet the forward derivability conditions;

[0013] Step 7: Based on the derivation starting point, perform backward derivation on each fragment element in a backward recursive manner to obtain the fragment elements that meet the backward derivability conditions;

[0014] Step 8: Forcefully assign values ​​to the fragment elements that meet the forward / backward derivability conditions through the assignment rules; the assignment item is the knowledge graph element corresponding to the current fragment element;

[0015] Among them, the derivation rules are used for forward derivation and backward derivation, and the large model is selectively used for auxiliary judgment. When the forward derivation condition or the backward derivation condition is met, the corresponding score element is removed from the continuity score array;

[0016] Step 9: Determine whether the number of loops has reached the limit or whether the continuity score array has been cleared; if so, end the loop and output the automatic association result; otherwise, return to step 5.

[0017] As a further solution, in step 1, content preprocessing is performed as follows:

[0018] Use the speech recognition service to perform speech recognition on the classroom recording video and obtain a speech recognition list with timestamps;

[0019] Intercept the speech recognition list by time period and generate a speaker id array; wherein the speaker id array is composed of speakers id and corresponding voice data;

[0020] Determine the speaker corresponding to the lecturer through recognition rules id , and output the voice data corresponding to the teacher to obtain the voice data of the teacher;

[0021] The teacher's speech data is segmented using a large language model to obtain a segment array.

[0022] As a further solution, the fragment element includes a fragment index, a start field, an end field, a title field, a description field, an embedded field, a knowledge graph matching index, a fragment score array and a score element; the knowledge graph array includes a knowledge graph index, a title field, a description field and an embedded field.

[0023] As a further solution, in step 3, the title field and description field are used to generate vectors for the fragment elements and knowledge graph elements respectively through the vector generation module, and the associated matching scores are scored by vector similarity. The associated matching scores below the specified threshold are deleted, and the preset top scores are retained from high to low. N Items are associated with matching scores and form an array of fragment scores.

[0024] As a further solution, in step 4, continuous cumulative scoring is performed through the following steps:

[0025] Determine the current fragment element , Previous fragment element and the next fragment element ;

[0026] Get the current fragment element Extract the corresponding knowledge graph elements from the fragment score array and form a knowledge point list ;in, List of knowledge points The maximum list number of;

[0027] Get the previous fragment element Extract the corresponding knowledge graph elements from the fragment score array and form a knowledge point list ;in, List of knowledge points The maximum list number of;

[0028] Get the next fragment element Extract the corresponding knowledge graph elements from the fragment score array and form a knowledge point list ;in, List of knowledge points The maximum list number of;

[0029] Calculate the current fragment element For the previous fragment element The cumulative score of the previous item continuity ;in, ;

[0030] Calculate the current fragment element For the next fragment element The cumulative score of the latter continuity ;in, ;

[0031] Calculate the current fragment element Continuous cumulative score ;in, .

[0032] As a further solution, in step 6, the judgment steps of the forward derivable conditions include:

[0033] The first forward derivable judgment: judge whether the knowledge graph element corresponding to the knowledge graph matching index of the fragment element where the forward iteration parameter is located is the same as the knowledge graph element corresponding to the fragment score array of the previous fragment element where the forward iteration parameter is located;

[0034] If so, the forward derivability condition is met, and the knowledge graph matching index of the previous fragment element is set as the knowledge graph matching index of the fragment element where the forward iteration parameter is located;

[0035] The second forward derivable judgment: judge whether the knowledge graph element corresponding to the previous knowledge graph matching index of the fragment element where the forward iteration parameter is located is the same as the knowledge graph element corresponding to the fragment score array of the previous fragment element where the forward iteration parameter is located;

[0036] If so, the forward derivability condition is met, and the knowledge graph matching index of the previous fragment element is set to the previous knowledge graph matching index of the fragment element where the forward iteration parameter is located;

[0037] Among them, if the first forward derivable judgment and the second forward derivable judgment are met at the same time, an auxiliary judgment is performed through the large model to determine the knowledge graph matching index of the previous fragment element.

[0038] As a further solution, in step 7, the judgment steps of the backward derivable conditions include:

[0039] The first backward derivable judgment: judge whether the knowledge graph matching index corresponding to the knowledge graph element of the fragment element where the backward iteration parameter is located is the same as the knowledge graph element corresponding to the fragment score array of the next fragment element where the backward iteration parameter is located;

[0040] If so, the backward derivability condition is met, and the knowledge graph matching index of the next fragment element is set to the knowledge graph matching index of the fragment element where the backward iteration parameter is located;

[0041] The second backward derivable judgment: judge whether the knowledge graph element corresponding to the next knowledge graph matching index of the fragment element where the backward iteration parameter is located is the same as the knowledge graph element corresponding to the fragment score array of the next fragment element where the backward iteration parameter is located;

[0042] If so, the backward derivable condition is met, and the knowledge graph matching index of the next fragment element is set to the next knowledge graph matching index of the fragment element where the backward iteration parameter is located;

[0043] Among them, if the first backward derivable judgment and the second backward derivable judgment are met at the same time, an auxiliary judgment is performed through the large model to determine the knowledge graph matching index of the next fragment element.

[0044] As a further solution, the derivation rules include forward derivation rules and backward derivation rules; among them,

[0045] Forward deduction rules:

[0046] Set the derivation starting point to the initial forward iteration parameters;

[0047] Perform a forward derivable conditional judgment;

[0048] After completing the forward derivation condition judgment, the current previous fragment element is set as the forward iteration parameter, and the new previous fragment element is determined to continue the forward derivation;

[0049] When three consecutive elements of the previous segment do not meet the forward derivation conditions, jump out of the forward derivation;

[0050] Backward deduction rules:

[0051] Set the derivation starting point to the initial backward iteration parameters;

[0052] Perform a backward derivable conditional judgment;

[0053] After completing the backward derivation condition judgment, the current next fragment element is set as the backward iteration parameter, and the new next fragment element is determined to continue the backward derivation;

[0054] When three consecutive next-segment elements do not meet the backward derivation condition, the backward derivation is exited.

[0055] As a further solution, the assignment rule:

[0056] The knowledge graph matching index of the previous fragment element is k And the knowledge graph matching index of the next fragment element is k , then the knowledge graph matching index of the current fragment element is k ;

[0057] The knowledge graph matching index of the previous fragment element is k And the knowledge graph matching index of the next fragment element is k +2, then the knowledge graph matching index of the current fragment element is k +1;

[0058] The knowledge graph matching index of the previous fragment element is k And the knowledge graph matching index of the next fragment element is k+1, then the large model is used to make auxiliary judgments and determine the knowledge graph matching index of the current fragment element is [ k,k +1] one of them.

[0059] On the other hand, the present invention also provides a system for automatically associating classroom video recordings with knowledge graphs, using a method for automatically associating classroom video recordings with knowledge graphs as described in any of the above items, including:

[0060] The first acquisition module: pre-processes the content of the classroom video to obtain a segment array;

[0061] The second acquisition module: obtains and generates the corresponding knowledge graph array based on the knowledge graph list;

[0062] The first scoring module: for each fragment element in the fragment array, associate and match the scores with all knowledge graph elements in the knowledge graph array, obtain the fragment score array and add it to the corresponding fragment element;

[0063] The second scoring module: performs a continuity cumulative score on the segment elements according to the segment score array and adds the score to the segment elements of the segment array, uses the continuity cumulative score of each segment element as a score element and generates a continuity score array;

[0064] Starting point setting module: queries the scoring element with the highest cumulative continuity score in the continuity score array, and sets the fragment element corresponding to the scoring element as the derivation starting point;

[0065] Forward derivation module: Based on the derivation starting point, each fragment element is forward deduced in a forward recursive manner to obtain the fragment elements that meet the forward derivability conditions;

[0066] Backward derivation module: Based on the derivation starting point, each fragment element is backward deduced in a recursive manner to obtain the fragment elements that meet the backward derivation conditions;

[0067] Rule assignment module: Forcefully assign values ​​to fragment elements that meet the forward / backward derivability conditions through assignment rules; the assignment item is the knowledge graph element corresponding to the current fragment element;

[0068] Iteration management module: determines whether the number of loops has reached the limit or whether the continuity score array has been cleared; if so, ends the loop and outputs the automatic association result; otherwise, returns to the starting point setting module;

[0069] Among them, during forward / backward deduction, the deduction rules are used for review, and the large model is selectively used for auxiliary judgment. When the forward / backward derivability conditions are met, the corresponding score elements are removed from the continuity score array.

[0070] Compared with related technologies, the method and system for automatically associating classroom recording videos with knowledge graphs provided by the present invention have the following advantages:

[0071] 1. This invention proposes a matching method that combines models and rules by comprehensively using vectors, large models, and the sequential relationship between knowledge graphs and teaching knowledge. Using only the speech recognition files of classroom recordings, it is possible to associate classroom recordings with knowledge graphs, quickly calculate the preceding and following related items, and complete knowledge point matching.

[0072] 2. The present invention utilizes the regular characteristic that under normal circumstances, there must be continuous segments in a lecture video belonging to the same knowledge point or continuous knowledge points. It adopts a derivation calculation that starts from a point and performs forward and backward knowledge point matching, realizing calculation from point to surface, thereby realizing the association between the entire video and the knowledge points. The sequential relationship of the lecture knowledge conforms to the objective distribution law;

[0073] 3. In the derivation process, the present invention combines knowledge point continuity rules with large models. Rules are used for association processing when not necessary, and large models are used for auxiliary judgment only when necessary. The setting of fixed rules reduces the amount of calculation, which can greatly improve efficiency, maintain the objectivity of the matching results, and avoid the influence of hallucinations to the greatest extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0076] Figure 1 This is an overall flow chart of a method for automatically associating classroom recording videos with knowledge graphs provided by the present invention;

[0077] Figure 2 This is a schematic diagram of the structure of an automatic association system for classroom recording videos and knowledge graphs provided by the present invention.

[0078] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0079] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0080] Example 1

[0081] See also Figure 1 , the embodiment of the present application provides a method for automatically associating classroom recording videos with knowledge graphs, including:

[0082] Step 1: Preprocess the content of the classroom video to obtain a segment array;

[0083] Step 2: Obtain and generate the corresponding knowledge graph array based on the knowledge graph list;

[0084] Step 3: For each fragment element in the fragment array, associate the matching scores with all knowledge graph elements in the knowledge graph array, obtain the fragment score array and add it to the corresponding fragment element;

[0085] Step 4: Calculate the continuity cumulative score of the fragment elements according to the fragment score array and add it to the fragment elements of the fragment array. Use the continuity cumulative score of each fragment element as the score element and generate a continuity score array.

[0086] Step 5: Query the score element with the highest cumulative continuity score in the continuity score array, and set the fragment element corresponding to the score element as the derivation starting point;

[0087] Step 6: Based on the derivation starting point, perform forward derivation on each fragment element in a forward recursive manner to obtain the fragment elements that meet the forward derivability conditions;

[0088] Step 7: Based on the derivation starting point, perform backward derivation on each fragment element in a backward recursive manner to obtain the fragment elements that meet the backward derivability conditions;

[0089] Step 8: Forcefully assign values ​​to the fragment elements that meet the forward / backward derivability conditions through the assignment rules; the assignment item is the knowledge graph element corresponding to the current fragment element;

[0090] Among them, the derivation rules are used for forward derivation and backward derivation, and the large model is selectively used for auxiliary judgment. When the forward derivation condition or the backward derivation condition is met, the corresponding score element is removed from the continuity score array;

[0091] Step 9: Determine whether the number of loops has reached the limit or whether the continuity score array has been cleared; if so, end the loop and output the automatic association result; otherwise, return to step 5.

[0092] It should be noted that both the current manual association method and the matching method through large models have their shortcomings; among them, the manual processing method has the problems of low efficiency, high cost, and subjectivity in matching selection, and there are also large fluctuations in accuracy; the matching method through pure large models is an improvement over the manual processing method, but it still has shortcomings in processing efficiency and accuracy, and the model is also affected by hallucinations, resulting in its matching results often contradictory, and the sequential relationship of the taught knowledge does not conform to the objective distribution law.

[0093] To this end, this embodiment proposes a matching method that combines models and rules by comprehensively using vectors, large models, and the sequential relationship between knowledge graphs and teaching knowledge. The association between classroom recording videos and knowledge graphs can be completed using only the voice recognition files of classroom recordings. This embodiment uses rules for association processing when it is not necessary, and only uses large models for auxiliary judgment when necessary. The setting of fixed rules can greatly improve efficiency, maintain the objectivity of matching results, and avoid the influence of hallucinations to the greatest extent.

[0094] In addition, this embodiment utilizes the rule that under normal circumstances there must be continuous segments in a teaching video belonging to the same knowledge point or continuous knowledge points, and adopts a deductive calculation starting from a point to perform forward and backward knowledge point matching, to realize calculation from point to surface, thereby realizing the association between the entire video and the knowledge points, and the sequential relationship of the teaching knowledge conforms to the objective distribution law.

[0095] In step 1, this embodiment preprocesses the classroom recording using speech recognition and speaker recognition. Speaker recognition removes content such as student discussions and presentations, resulting in teacher speech recognition data. A large model is used to generate segments from the teacher speech recognition data, with a granularity of 3-5 minutes per segment. Each segment has a title and description.

[0096] Specifically, content preprocessing is performed through the following steps:

[0097] Use the speech recognition service to perform speech recognition on the classroom recording video and obtain a speech recognition list with timestamps;

[0098] The speech recognition list is set as follows (in milliseconds):

[0099] 0-4000 : Statement 1

[0100] 4000-8000: Statement 2

[0101] Intercept the speech recognition list by time period and generate a speaker id array; wherein the speaker id array is composed of speakers id and corresponding voice data;

[0102] Among them, the speaker id array format is [ speaker 1 ,speaker 2 ,speaker 3 .... ];

[0103] Determine the speaker corresponding to the lecturer through recognition rules id , and output the voice data corresponding to the teacher to obtain the voice data of the teacher;

[0104] In current educational scenarios, the teacher in a video must be the person who speaks the most in the first and last three minutes of the video. This is used as the recognition rule to identify the teacher.

[0105] The language model is used to generate segments of the teacher's speech data to obtain a segment array, and its array format is the same as the speaker ID array.

[0106] Furthermore, a fragment array ( ) , Contains several fragment elements ( ), and the fragment element includes the fragment index ( s_index ), Start Field ( begin ), end field ( end ), title field ( title ), description field ( desc ), embedded fields ( embedding ), Knowledge Graph Matching Index ( match_kg_index ); In addition, a subsequent step adds an array of fragment scores to the fragment element ( ) and the score element ( ); while the knowledge graph array includes the knowledge graph index ( k_index ), title field ( title ), description field ( desc ) and embedded fields ( embedding ).

[0107] On this basis, this embodiment uses the title field and the description field to generate vectors for the fragment elements and the knowledge graph elements respectively through the vector generation module, and performs association matching scores based on vector similarity, deletes the association matching scores below the specified threshold, and retains the preset top scores from high to low. N Items are associated with matching scores and form an array of fragment scores.

[0108] In step 4, this embodiment uses the correlation between the teaching content and the context of the knowledge point list to calculate a probability score for each segment, forming an independent segment score array for data selection. Specifically, continuous cumulative scoring is performed through the following steps:

[0109] Determine the current fragment element , Previous fragment element and the next fragment element ;

[0110] Get the current fragment element Extract the corresponding knowledge graph elements from the fragment score array and form a knowledge point list ;in, List of knowledge points The maximum list number of;

[0111] Get the previous fragment element Extract the corresponding knowledge graph elements from the fragment score array and form a knowledge point list ;in, List of knowledge points The maximum list number of;

[0112] Get the next fragment element Extract the corresponding knowledge graph elements from the fragment score array and form a knowledge point list ;in, List of knowledge points The maximum list number of;

[0113] Calculate the current fragment element For the previous fragment element The cumulative score of the previous item continuity ;in, ;

[0114] Calculate the current fragment element For the next fragment element The cumulative score of the latter continuity ;in, ;

[0115] Calculate the current fragment element Continuous cumulative score ;in, .

[0116] in, Indicates the first layer of loop (loop of all knowledge points of the current item), Indicates the second layer of loop (loop of all knowledge points in the next item), 1 indicates the standard score, Used to determine whether the knowledge points are equal. Indicates or conditions, Used to determine whether the knowledge point differs by -1. Used to determine whether the knowledge points differ by +1.

[0117] This fragment element The inner element is filled with ( s_index,begin,end ,title , desc, embedding , match_kg_index , , ), and generates a Non-zero continuity score array ScoreArray, Its internal elements are ( s_index,score ).

[0118] It should be noted that: this embodiment obtains the scoring element with the highest cumulative continuity score from the fragment score array in step 4, and matches and confirms the associated knowledge points through the large model. After the matching is completed, the fragment of the entry is used as the starting point, and the forward and backward derivations of the knowledge points are performed on the forward and backward fragments of the fragment array. The successfully derived nodes are deleted from the continuity score array. After each round of derivation, the data is sorted using rules; the above process is repeated until the continuity score array is empty or the end condition of the loop is reached.

[0119] In a specific embodiment, by looping through ScoreArray The element with the highest score in the is used as the starting point ( currentpoint ), the corresponding pseudo code is:

[0120] loopcount = len ( ScoreArray )

[0121] whilelen ( ScoreArray ) and loopcount!= 0 :

[0122] currentpoint = find_max_score () # Find the largest fragment element

[0123] # There are multiple knowledge points at the current point, and the large model judges

[0124] matchkgindex = llm_adjust_kg_index ( currentpoint )

[0125] forward ( currentpoint ) # Forward derivation

[0126] backword (currentpoint ) # Backward deduction

[0127] rule_adjust () # Rule derivation

[0128] loopcount =loopcount -1

[0129] In the pseudocode, the deduction rules include forward deduction rules and backward deduction rules; among them,

[0130] Forward deduction rules:

[0131] Set the derivation starting point to the initial forward iteration parameters;

[0132] Perform a forward derivable conditional judgment;

[0133] After completing the forward derivation condition judgment, the current previous fragment element is set as the forward iteration parameter, and the new previous fragment element is determined to continue the forward derivation;

[0134] When three consecutive elements of the previous segment do not meet the forward derivation conditions, jump out of the forward derivation;

[0135] Backward deduction rules:

[0136] Set the derivation starting point to the initial backward iteration parameters;

[0137] Perform a backward derivable conditional judgment;

[0138] After completing the backward derivation condition judgment, the current next fragment element is set as the backward iteration parameter, and the new next fragment element is determined to continue the backward derivation;

[0139] When three consecutive next-segment elements do not meet the backward derivation condition, the backward derivation is exited.

[0140] More specifically, in step 6, the steps of determining the forward derivable condition include:

[0141] The first forward derivable judgment: judge the fragment element where the forward iteration parameter is located ( )’s knowledge graph matching index ( match_kg_index ) corresponds to the knowledge graph element, whether it is the same as the previous fragment element where the forward iteration parameter is located ( )'s fragment scores array ( ) The corresponding knowledge graph elements are the same;

[0142] If so, the forward derivation condition is satisfied, and the previous fragment element ( )’s knowledge graph matching index is set to the fragment element where the forward iteration parameter is located ( )’s knowledge graph matching index ( match_kg_index );

[0143] The second forward derivable judgment: judge the fragment element where the forward iteration parameter is located ( )'s previous knowledge graph matching index ( match_kg_index -1) corresponds to the knowledge graph element, whether it is consistent with the previous fragment element where the forward iteration parameter is located ( )'s fragment scores array ( ) The corresponding knowledge graph elements are the same;

[0144] If so, the forward derivable condition is met, and the knowledge graph matching index of the previous fragment element is set to the previous knowledge graph matching index of the fragment element where the forward iteration parameter is located ( match_kg_index -1);

[0145] Among them, if the first forward derivable judgment and the second forward derivable judgment are satisfied at the same time, the large model is used to make auxiliary judgments to determine the knowledge graph matching index of the previous fragment element, that is, to select ( match_kg_index -1) or ( match_kg_index ).

[0146] More specifically, in step 7, the steps for determining the backward derivable condition include:

[0147] The first backward derivation judgment: judge the fragment element where the backward iteration parameter is located ( )’s knowledge graph matching index ( match_kg_index ) corresponds to the knowledge graph element, whether it is the next fragment element where the backward iteration parameter is located ( )'s fragment scores array ( ) The corresponding knowledge graph elements are the same;

[0148] If so, the backward derivable condition is met, and the knowledge graph matching index of the next fragment element is set to the knowledge graph matching index of the fragment element where the backward iteration parameter is located ( match_kg_index );

[0149] The second backward derivation judgment: judge the fragment element where the backward iteration parameter is located ( )'s next knowledge graph matching index ( match_kg_index +1) corresponds to the knowledge graph element, whether it is consistent with the next fragment element where the backward iteration parameter is located ( )'s fragment scores array ( ) The corresponding knowledge graph elements are the same;

[0150] If so, the backward derivable condition is met, and the knowledge graph matching index of the next fragment element is set to the next knowledge graph matching index of the fragment element where the backward iteration parameter is located ( match_kg_index +1);

[0151] Among them, if the first backward derivable judgment and the second backward derivable judgment are satisfied at the same time, the large model is used to make auxiliary judgments to determine the knowledge graph matching index of the next segment element; that is, select ( match_kg_index ) or ( match_ kg_index +1).

[0152] Specifically, the assignment rules are:

[0153] The knowledge graph matching index of the previous fragment element is k And the knowledge graph matching index of the next fragment element is k , then the knowledge graph matching index of the current fragment element is k ;

[0154] The knowledge graph matching index of the previous fragment element is k And the knowledge graph matching index of the next fragment element is k +2, then the knowledge graph matching index of the current fragment element is k +1;

[0155] The knowledge graph matching index of the previous fragment element is k And the knowledge graph matching index of the next fragment element is k +1, then the large model is used to make auxiliary judgments and determine the knowledge graph matching index of the current fragment element is [ k,k +1] one of them.

[0156] Example 2

[0157] See also Figure 2 This embodiment further provides a system for automatically associating classroom video recordings with knowledge graphs, using a method for automatically associating classroom video recordings with knowledge graphs as described in Example 1, including:

[0158] The first acquisition module: pre-processes the content of the classroom video to obtain a segment array;

[0159] The second acquisition module: obtains and generates the corresponding knowledge graph array based on the knowledge graph list;

[0160] The first scoring module: for each fragment element in the fragment array, associate and match the scores with all knowledge graph elements in the knowledge graph array, obtain the fragment score array and add it to the corresponding fragment element;

[0161] The second scoring module: performs a continuity cumulative score on the segment elements according to the segment score array and adds the score to the segment elements of the segment array, uses the continuity cumulative score of each segment element as a score element and generates a continuity score array;

[0162] Starting point setting module: queries the scoring element with the highest cumulative continuity score in the continuity score array, and sets the fragment element corresponding to the scoring element as the derivation starting point;

[0163] Forward derivation module: Based on the derivation starting point, each fragment element is forward deduced in a forward recursive manner to obtain the fragment elements that meet the forward derivability conditions;

[0164] Backward derivation module: Based on the derivation starting point, each fragment element is backward deduced in a recursive manner to obtain the fragment elements that meet the backward derivation conditions;

[0165] Rule assignment module: Forcefully assign values ​​to fragment elements that meet the forward / backward derivability conditions through assignment rules; the assignment item is the knowledge graph element corresponding to the current fragment element;

[0166] Iteration management module: determines whether the number of loops has reached the limit or whether the continuity score array has been cleared; if so, ends the loop and outputs the automatic association result; otherwise, returns to the starting point setting module;

[0167] Among them, during forward / backward deduction, the deduction rules are used for review, and the large model is selectively used for auxiliary judgment. When the forward / backward derivability conditions are met, the corresponding score elements are removed from the continuity score array.

[0168] This embodiment is based on the system provided in Example 1, and its function is the same as that of Example 1. Both of them propose a matching method that combines models and rules by comprehensively using vectors, large models, and the sequential relationship between knowledge graphs and teaching knowledge. The association between classroom recording videos and knowledge graphs can be completed using only the voice recognition files of classroom recordings. This embodiment uses rules for association processing when it is not necessary, and only uses large models for auxiliary judgment when necessary. The setting of fixed rules can greatly improve efficiency, maintain the objectivity of matching results, and avoid the influence of hallucinations to the greatest extent.

[0169] In the specific use process, through the description method of the present invention, by combining the 0.6B vector model and the 14B level large model in the knowledge graph association scenario, a higher accuracy matching can be achieved and the manual matching cost can be reduced.

[0170] The above are only some embodiments of the present application and are not intended to limit the patent scope of the present application. All equivalent structural transformations made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for automatically associating classroom recording videos with knowledge graphs, characterized in that: include: Step 1: Preprocess the content of the classroom video to obtain a segment array; Step 2: Obtain and generate the corresponding knowledge graph array based on the knowledge graph list; Step 3: For each fragment element in the fragment array, associate the matching scores with all knowledge graph elements in the knowledge graph array, obtain the fragment score array and add it to the corresponding fragment element; Step 4: Calculate the continuity cumulative score of the fragment elements according to the fragment score array and add it to the fragment elements of the fragment array. Use the continuity cumulative score of each fragment element as the score element and generate a continuity score array. Step 5: Query the score element with the highest cumulative continuity score in the continuity score array, and set the fragment element corresponding to the score element as the derivation starting point; Step 6: Based on the derivation starting point, perform forward derivation on each fragment element in a forward recursive manner to obtain the fragment elements that meet the forward derivability conditions; Step 7: Based on the derivation starting point, perform backward derivation on each fragment element in a backward recursive manner to obtain the fragment elements that meet the backward derivability conditions; Step 8: Forcefully assign values ​​to the fragment elements that meet the forward / backward derivability conditions through the assignment rules; the assignment item is the knowledge graph element corresponding to the current fragment element; Among them, the derivation rules are used for forward derivation and backward derivation, and the large model is selectively used for auxiliary judgment. When the forward derivation condition or the backward derivation condition is met, the corresponding score element is removed from the continuity score array; Step 9: Determine whether the number of loops has reached the limit or whether the continuity score array has been cleared; if so, end the loop and output the automatic association result; otherwise, return to step 5.

2. The method for automatically associating classroom video recordings with knowledge graphs according to claim 1 is characterized in that: In step 1, content preprocessing is performed through the following steps: Use the speech recognition service to perform speech recognition on the classroom recording video and obtain a speech recognition list with timestamps; Intercept the speech recognition list by time period and generate a speaker id array; wherein the speaker id array is composed of speakers id and corresponding voice data; Determine the speaker corresponding to the lecturer through recognition rules id , and output the voice data corresponding to the teacher to obtain the voice data of the teacher; The teacher's speech data is segmented using a large language model to obtain a segment array.

3. The method for automatically associating classroom video recordings with knowledge graphs according to claim 1 is characterized in that: The fragment element includes a fragment index, a start field, an end field, a title field, a description field, an embedded field, a knowledge graph matching index, a fragment score array and a score element; the knowledge graph array includes a knowledge graph index, a title field, a description field and an embedded field.

4. The method for automatically associating classroom video recordings with knowledge graphs according to claim 3 is characterized in that: In step 3, the title field and description field are used to generate vectors for the fragment elements and knowledge graph elements respectively through the vector generation module, and the associated matching scores are scored by vector similarity. The associated matching scores below the specified threshold are deleted, and the preset top scores are retained from high to low. N Items are associated with matching scores and form an array of fragment scores.

5. The method for automatically associating classroom video recordings with knowledge graphs according to claim 1 is characterized in that: In step 4, continuous cumulative scoring is performed through the following steps: Determine the current fragment element , Previous fragment element and the next fragment element ; Get the current fragment element Extract the corresponding knowledge graph elements from the fragment score array and form a knowledge point list ;in, List of knowledge points The maximum list number of; Get the previous fragment element Extract the corresponding knowledge graph elements from the fragment score array and form a knowledge point list ;in, List of knowledge points The maximum list number of; Get the next fragment element Extract the corresponding knowledge graph elements from the fragment score array and form a knowledge point list ;in, List of knowledge points The maximum list number of; Calculate the current fragment element For the previous fragment element The cumulative score of the previous item continuity ;in, ; Calculate the current fragment element For the next fragment element The cumulative score of the latter continuity ;in, ; Calculate the current fragment element Continuous cumulative score ;in, .

6. The method for automatically associating classroom video recordings with knowledge graphs according to claim 1, characterized in that: In step 6, the steps for determining the forward derivable condition include: The first forward derivable judgment: judge whether the knowledge graph element corresponding to the knowledge graph matching index of the fragment element where the forward iteration parameter is located is the same as the knowledge graph element corresponding to the fragment score array of the previous fragment element where the forward iteration parameter is located; If so, the forward derivability condition is met, and the knowledge graph matching index of the previous fragment element is set as the knowledge graph matching index of the fragment element where the forward iteration parameter is located; The second forward derivable judgment: judge whether the knowledge graph element corresponding to the previous knowledge graph matching index of the fragment element where the forward iteration parameter is located is the same as the knowledge graph element corresponding to the fragment score array of the previous fragment element where the forward iteration parameter is located; If so, the forward derivability condition is met, and the knowledge graph matching index of the previous fragment element is set to the previous knowledge graph matching index of the fragment element where the forward iteration parameter is located; Among them, if the first forward derivable judgment and the second forward derivable judgment are met at the same time, an auxiliary judgment is performed through the large model to determine the knowledge graph matching index of the previous fragment element.

7. The method for automatically associating classroom video recordings with knowledge graphs according to claim 1, characterized in that: In step 7, the steps for determining the backward derivable condition include: The first backward derivable judgment: judge whether the knowledge graph matching index corresponding to the knowledge graph element of the fragment element where the backward iteration parameter is located is the same as the knowledge graph element corresponding to the fragment score array of the next fragment element where the backward iteration parameter is located; If so, the backward derivability condition is met, and the knowledge graph matching index of the next fragment element is set to the knowledge graph matching index of the fragment element where the backward iteration parameter is located; The second backward derivable judgment: judge whether the knowledge graph element corresponding to the next knowledge graph matching index of the fragment element where the backward iteration parameter is located is the same as the knowledge graph element corresponding to the fragment score array of the next fragment element where the backward iteration parameter is located; If so, the backward derivable condition is met, and the knowledge graph matching index of the next fragment element is set to the next knowledge graph matching index of the fragment element where the backward iteration parameter is located; Among them, if the first backward derivable judgment and the second backward derivable judgment are met at the same time, an auxiliary judgment is performed through the large model to determine the knowledge graph matching index of the next fragment element.

8. The method for automatically associating classroom video recordings with knowledge graphs according to claim 6 or claim 7, characterized in that: The deduction rules include forward deduction rules and backward deduction rules; among them, Forward deduction rules: Set the derivation starting point to the initial forward iteration parameters; Perform a forward derivable conditional judgment; After completing the forward derivation condition judgment, the current previous fragment element is set as the forward iteration parameter, and the new previous fragment element is determined to continue the forward derivation; When three consecutive elements of the previous segment do not meet the forward derivation conditions, jump out of the forward derivation; Backward deduction rules: Set the derivation starting point to the initial backward iteration parameters; Perform a backward derivable conditional judgment; After completing the backward derivation condition judgment, the current next fragment element is set as the backward iteration parameter, and the new next fragment element is determined to continue the backward derivation; When three consecutive next-segment elements do not meet the backward derivation condition, the backward derivation is exited.

9. The method for automatically associating classroom video recordings with knowledge graphs according to claim 8, characterized in that: Assignment rules: The knowledge graph matching index of the previous fragment element is k And the knowledge graph matching index of the next fragment element is k , then the knowledge graph matching index of the current fragment element is k ; The knowledge graph matching index of the previous fragment element is k And the knowledge graph matching index of the next fragment element is k +2, then the knowledge graph matching index of the current fragment element is k +1; The knowledge graph matching index of the previous fragment element is k And the knowledge graph matching index of the next fragment element is k +1, then the large model is used to make auxiliary judgments and determine the knowledge graph matching index of the current fragment element is [ k,k +1] one of them.

10. A system for automatically associating classroom video recordings with knowledge graphs, using a method for automatically associating classroom video recordings with knowledge graphs as described in any one of claims 1 to 9, characterized in that: include: The first acquisition module: pre-processes the content of the classroom video to obtain a segment array; The second acquisition module: obtains and generates the corresponding knowledge graph array based on the knowledge graph list; The first scoring module: for each fragment element in the fragment array, associate and match the scores with all knowledge graph elements in the knowledge graph array, obtain the fragment score array and add it to the corresponding fragment element; The second scoring module: performs a continuity cumulative score on the segment elements according to the segment score array and adds the score to the segment elements of the segment array, uses the continuity cumulative score of each segment element as a score element and generates a continuity score array; Starting point setting module: queries the scoring element with the highest cumulative continuity score in the continuity score array, and sets the fragment element corresponding to the scoring element as the derivation starting point; Forward derivation module: Based on the derivation starting point, each fragment element is forward deduced in a forward recursive manner to obtain the fragment elements that meet the forward derivability conditions; Backward derivation module: Based on the derivation starting point, each fragment element is backward deduced in a recursive manner to obtain the fragment elements that meet the backward derivation conditions; Rule assignment module: Forcefully assign values ​​to fragment elements that meet the forward / backward derivability conditions through assignment rules; the assignment item is the knowledge graph element corresponding to the current fragment element; Iteration management module: determines whether the number of loops has reached the limit or whether the continuity score array has been cleared; if so, ends the loop and outputs the automatic association result; otherwise, returns to the starting point setting module; Among them, during forward / backward deduction, the deduction rules are used for review, and the large model is selectively used for auxiliary judgment. When the forward / backward derivability conditions are met, the corresponding score elements are removed from the continuity score array.

Images