Video content-oriented knowledge base construction method and system

Abstract

The invention provides a video content-oriented knowledge base construction method and system. The method comprises the steps of text acquisition, text alignment, multi-source text fusion, semantic calibration, knowledge unit generation and the like. According to the method, a voice recognition text and a subtitle text of a video are acquired, and a fusion weight is calculated through theme keywords, semantic similarity and context consistency, so that dynamic weighted fusion of multi-source texts is realized. The invention further provides a fusion confidence coefficient stability correction mechanism of cross-time backtracking, a fusion weight sequence of continuous fragments is recorded, a stability score is calculated, backtracking correction is triggered when abnormal fluctuation of the confidence coefficient is detected, semantic feature extraction and weight calculation are carried out on the nearest fragments again, and the stability score is calculated. Therefore, short-time noise and local error accumulation are inhibited. Through the processing, the text fusion stability and the overall semantic consistency in the long video scene can be remarkably improved, and the accuracy of knowledge unit extraction and structured expression is improved. The method is suitable for the fields of large-scale video content management, intelligent document production, education, media knowledge base construction and the like.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence and natural language processing, and particularly relates to a knowledge base construction method and system for video content. BACKGROUND

[0002] With the rapid growth of multimedia content, videos have become an important carrier of knowledge dissemination. Especially in the fields of online education (MOOC), medical training and enterprise knowledge management, a large number of teaching videos, explanation videos and conference videos are continuously accumulated. However, these video data mainly exist in the form of unstructured voice, subtitles and the like, and do not have direct knowledge retrieval or question and answer functions.

[0003] At present, video knowledge extraction usually goes through three stages of "speech recognition-text extraction-knowledge extraction". However, the existing technology has the following problems: although an automatic speech recognition (ASR) system can transcribe audio into text, in the presence of accents, noise interference, professional terms and the like, the recognition error rate is high, and especially in professional fields such as medicine, engineering and education, it is difficult to guarantee the text quality simply by relying on ASR. In addition, videos often contain subtitle files (such as SRT, VTT formats), and current video content knowledge processing mainly relies on the following technical paths: only keyword indexing based on ASR text, lacking semantic understanding ability; relying on manual annotation or manual abstract, high cost and unable to scale; the existing video retrieval methods are mainly based on time axis positioning, and do not have knowledge level retrieval ability. The above methods cannot realize the automatic conversion from video to structured knowledge unit, especially lack the ability of semantic fusion and theme clustering across text sources, and it is difficult to meet the needs of intelligent question and answer and deep knowledge retrieval.

[0004] Therefore, there is an urgent need for a technical solution that can significantly improve the accuracy of speech-to-text, realize semantic clustering and knowledge unit expression, so as to construct a video knowledge base that can be used for intelligent retrieval and question and answer. SUMMARY

[0005] The application aims to provide a knowledge base construction method for video content, which comprises the following steps: S1, acquiring an input video and extracting an audio stream and a subtitle text; S2, recognizing the audio stream by using a speech recognition model to generate a first text sequence; S3, matching the first text sequence with the subtitle text based on a timestamp alignment strategy, and further vectorizing the matching segments by using a semantic embedding model to calculate segment-level semantic similarity; S4, realizing dynamic fusion of multi-source texts according to the similarity, and outputting a second text sequence; S5. Clustering the semantic segments of the second text sequence to obtain a plurality of semantic topic clusters; S6. Performing feature extraction on each semantic topic cluster to generate a structured knowledge unit; S7. Vectorizing the knowledge unit and storing it into a vector database for subsequent retrieval calls.

[0006] Preferably, in step S3, the timestamp alignment strategy includes: when one ASR segment corresponds to multiple subtitle segments, splicing the multiple subtitle segments to be the matching object; when multiple ASR segments correspond to one subtitle segment, merging the multiple ASR segments to be the matching object.

[0007] Preferably, in step S4, the dynamic fusion of the multi-source text according to the similarity includes: calculating a fusion weight α based on the semantic similarity Sim, wherein wherein β0, β1 are preset coefficients, and the first text sequence and the subtitle text are weighted and interpolated by α to generate the second text sequence.

[0008] Preferably, in step S4, it further includes context consistency correction, specifically including: When the context similarity of the current fusion segment and the previous fusion segment is lower than a set threshold, dynamically adjusting the fusion weight α according to the theme keywords of the historical segment, and regenerating the calibrated text.

[0009] Preferably, in step S5, the clustering of the semantic segments of the second text sequence includes: performing semantic segmentation on the semantic segments of the second text sequence, performing initial segmentation according to punctuation, and calculating the semantic similarity of adjacent segments, merging the segments when the similarity is greater than a preset value, and the length of a single segment does not exceed a preset number of characters.

[0010] Preferably, in step S6, the structured knowledge unit includes: theme, abstract text and keyword set, wherein the abstract text is generated by the TextRank algorithm.

[0011] Another aspect of the present application is to provide a cross-time backtracking stability correction method for multi-source fusion of video text, characterized in that it comprises: S1. Obtaining the multi-source text corresponding to the video segment, including the speech recognition text and the subtitle text, calculating the fusion weight based on the semantic features of the two , obtaining the current fusion text segment ; S2. Writing the fusion weight into a fusion weight sequence arranged in chronological order and extract the fusion weights of the last K segments within a preset time window K: ; S3. Based on the fusion weight sequence Calculate the stability score The stability score includes: ; Wherein, represents the sequence variance, is the context similarity sequence corresponding to the segment, and λ is the stability adjustment coefficient; S4. Compare the stability score with a preset threshold When , trigger the backtracking correction operation, and perform re-fusion calculation on the last fusion segments; S5. Recalculate the fusion weights of the last fusion segments based on the updated semantic features, context consistency and historical theme keywords to , and generate a backtracking corrected fusion text sequence to improve the temporal stability and global semantic consistency of text fusion; wherein the backtracking correction is a replacement of the original .

[0012] The application also provides a knowledge base construction system for video content, which comprises an audio and video preprocessing module, a speech recognition module, a semantic calibration module, a semantic clustering module, a knowledge unit construction module and a knowledge base calling module. The audio and video preprocessing module is used for extracting audio streams and subtitle texts. The speech recognition module is used for generating a first text sequence according to the audio streams. The semantic calibration module is used for performing timestamp alignment, semantic similarity calculation, adaptive fusion and language model generation, and outputting a calibrated second text sequence. The semantic clustering module is used for clustering semantic segments of the second text sequence to obtain semantic theme clusters. The knowledge unit construction module is used for generating structured knowledge units according to the semantic theme clusters, and vectorizing and storing the knowledge units into a vector database. The knowledge base calling module is based on the vector database and is used for subsequent retrieval and calling.

[0013] Preferably, the system further comprises a semantic segmentation module; the semantic segmentation module is configured to perform text segmentation on the first text sequence according to semantic boundaries to obtain semantic segments of the second text sequence.

[0014] Preferably, the semantic calibration module comprises a time alignment unit, a similarity calculation unit and a weighted fusion unit.

[0015] Preferably, the semantic calibration module further comprises a context consistency correction unit, which is configured to dynamically adjust the fusion weight a according to the theme keyword of the historical segment and regenerate the calibration text when the context similarity between the current fusion segment and the previous fusion segment is lower than a set threshold.

[0016] The present application adopts a text extraction mechanism based on acoustic channel (ASR) and subtitle semantic calibration. The mechanism performs dynamic fusion and calibration between ASR recognition results and video subtitles through timestamp synchronization and semantic similarity calculation, thereby correcting ASR errors. The mechanism simultaneously utilizes ASR and visual text channel (subtitles); calculates text segment similarity through a semantic embedding model (such as BERT); and adaptively selects the version with higher semantic consistency in the conflict segment. Compared with single ASR recognition, the present application can improve the transcription accuracy by 20% to 30% in noisy environments and term-intensive scenarios, and significantly reduce semantic deviation.

[0017] In addition, a knowledge unit construction method based on semantic clustering is adopted. The calibration text is grouped according to semantic similarity through vectorization and clustering algorithms (such as cosine similarity clustering, K-Means or hierarchical clustering), and then core knowledge points are extracted through semantic compression to form knowledge units in a unified format. Compared with the traditional full-text indexing method, the knowledge retrieval accuracy is improved by about 35%, the knowledge compression rate is more than 60%, and the knowledge reuse and retrieval efficiency is significantly improved.

[0018] In order to further improve the stability and overall semantic consistency of multi-source text fusion, the present application further proposes a fusion confidence sequence stability correction mechanism for cross-time backtracking. The mechanism records the time sequence of the fusion weight a at each time, and combines the context semantic consistency score to analyze the trend of the fusion confidence of the multi-frame text segment. When abnormal fluctuations, mutations or unstable patterns are detected within a continuous time window, the system triggers cross-segment backtracking correction, and re-performs weighted calculation on the previous several fusion segments to eliminate local errors caused by temporary noise, incorrect subtitles or short-term ASR deviation. The present mechanism can ensure stable output of text calibration at the sequence level, thereby significantly improving the accuracy and consistency of the final knowledge unit construction. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1is a video content-oriented knowledge base construction system structure provided by the present application. Figure 2 is a video content-oriented knowledge base construction flowchart provided by the present application. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. The technical solutions in the embodiments of the present application are intended to cover all possible modifications. For example, the processes, methods, systems, products or devices that include a series of steps or units are not limited to the listed steps or units, but can optionally include steps or units that are not listed, or can optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0021] Embodiment one: The overall structure of the system provided by the present application is shown in Figure 1 , preferably comprising the following functional modules: Audio and video preprocessing module 10: responsible for extracting the audio stream and subtitle stream of the video. This module is responsible for the analysis of the input video. In a preferred implementation, FFmpeg or a multimedia interface can be called to output the audio file A(t) and the subtitle file S(t), and they are transmitted to the ASR recognition module 20 and the semantic calibration module 30, respectively.

[0022] ASR recognition module 20: transcribe the received audio file A(t) stream into text. This module preferably uses an end-to-end speech recognition network (such as Whisper or Conformer) to decode the audio stream and generate the initial text T_ASR.

[0023] Semantic calibration module 30: used for multi-dimensional semantic fusion of speech recognition text and subtitle text to improve the accuracy of the transcription result. This module receives the subtitle file S(t) from the audio and video preprocessing module 10 and the initial writing text T_ASR from the ASR recognition module 20. Through timestamp matching and semantic similarity calculation, dynamic fusion of multi-source text is realized, and high-precision text T_final is output.

[0024] In a preferred embodiment, the semantic calibration module 30 can include the following three sub-units: Time alignment unit 31: for segment-level synchronization matching of ASR text and subtitle text according to timestamp information, to determine the possible corresponding text pairs. In a preferred embodiment, to deal with the non-strict correspondence of subtitles and ASR in the time dimension, the application adopts the following extended alignment strategy: if an ASR segment time window covers multiple subtitle segments, the subtitle content is automatically spliced to form Sj'. If one subtitle segment corresponds to multiple short ASR segments, the ASR segments are combined into a window and matched with the subtitle segment. If there is a missing on one side (such as missing subtitles), the single-source text is used for subsequent fusion, and no double-source fusion is performed.

[0025] Similarity calculation unit 32: for calculating the semantic similarity between corresponding segments based on a semantic embedding model (such as BERT), and outputting a similarity matrix . Wherein i represents the i-th segment from the ASR recognized text sequence, and j represents the j-th segment from the subtitle text sequence. Each segment is encoded into a semantic vector and E(Sj) .

[0026] In a preferred embodiment, the elements of the similarity matrix are defined as: ; Wherein E() represents a semantic embedding function, which is used to map the input text x into a high-dimensional semantic vector.

[0027] The specific implementation of the semantic embedding model E() used by the application is as follows: for Chinese text, a Sentence-BERT (SBERT) Chinese pre-trained model is preferably used, the model structure is a double-tower Transformer encoder, and the parameter size is about 110M. The text input is first segmented (BERT WordPiece Tokenizer is used), and the maximum length is 256 tokens. The sentence vector generation method is Mean Pooling, that is, the average of all token hidden vectors is taken as the output vector. The output vector dimension is fixed at 768. To adapt to the sentence style in the video field, the application can optionally fine-tune on 100,000 online course and subtitle corpora, and the fine-tuning task is similarity learning (SimCSE).

[0028] T_ASRi represents the i-th text segment recognized by ASR; Sj represents the j-th segment in the subtitle file i . jThe matrix represents the semantic similarity between each ASR segment and all caption segments. For example, assuming the system divides the ASR output text into 3 segments and the caption text into 3 segments, the calculated similarity matrix can be exemplified as follows: In this matrix, Sim(1,1)=0.92 indicates that the semantic similarity between T_ASR1 and subtitle segment S1 is 92%, and the system determines that the two are matching segments based on this. The similarity calculation unit outputs the matrix result to the semantic decision unit 33 for subsequent weighted fusion and text correction.

[0029] Semantic decision unit 33: used to perform weighted fusion based on similarity threshold and contextual consistency rules, select the text version with higher semantic consistency, and generate the final corrected text fragment.

[0030] This unit receives a similarity matrix from the similarity calculation unit 32. And according to each ASR fragment With corresponding subtitles The similarity value determines the fusion strategy. Its core process includes the following three steps: 1) Threshold determination: Set a semantic similarity threshold θ (e.g., 0.8).

[0031] when When two segments are considered to have the same semantic meaning, the subtitle text Sj is preferred.

[0032] when If a semantic conflict is detected, the process proceeds to the weighted fusion step.

[0033] 2) Weighted fusion calculation: For semantically conflicting segments, perform similarity-based adaptive weighted fusion to generate corrected text. T_fused_ i .

[0034] The calculation formula is as follows: ; Where α is the adaptive weight, which depends on the similarity. ; When Sim(i,j) is high, such as 0.7≤Sim(i,j)<θ (θ=0.95), the credibility of ASR text is high, and α takes a larger value (such as 0.8). When Sim(i,j) is low, such as Sim(i,j)<0.7, the credibility of the subtitle text is high, and α takes a smaller value (such as 0.3).

[0035] The specific calculation of a can be represented as, for example: ; Conventionally, , 0.3 is a constant term β0 in the adaptive weight function, used to ensure that the ASR text still has the lowest weight proportion when the semantic similarity is low; 0.5 is a coefficient β1, used to dynamically adjust the fusion proportion according to the similarity.

[0036] The weighted operation is implemented through word-level or sentence-level semantic interpolation as follows: the system represents the embedding vectors of the two texts as and , respectively, and calculates the fused vector as ; The fused vector is then decoded into a natural language text segment by a language generation model 3) Context consistency correction: To avoid semantic mutations between segments, the system calculates the context similarity between the current fused segment and the previous segment .When , the context correction mechanism is triggered: the value of a is adjusted according to the context topic words (such as keywords or topic labels), so that the text semantics remain coherent.

[0037] In a preferred embodiment, the semantic decision unit 33 calculates the context similarity between the current fused segment and the previous fused segment after completing the weighted fusion, for determining semantic coherence. The context similarity can be calculated as the cosine similarity between the semantic vectors of the two segments: where: E() is the semantic embedding function; its value range is [0, 1], and the closer to 1, the more consistent the semantics. When Sim_ctx< γ, the system determines that there is a semantic mutation, where γ is the context consistency threshold, preferably 0.7.

[0038] After triggering the correction, the system performs the following two steps: Step 1: Context topic word extraction The system performs keyword extraction (such as TextRank or TF-IDF) on the previous segment to obtain a set of context topic words .

[0039] Step 2: Adaptive weight recalculation To enhance semantic continuity, the system increases the weight of the context-related text, and the formula is revised as: ; Wherein: is the revised fusion weight; is an adjustment parameter, preferably 0.2-0.4; the lower is, the greater the correction range is.

[0040] For example, the previous fusion segment is: “Matrix multiplication is essentially a combination of linear transformations.” The current fusion segment candidate text is: ASR text: “We are now talking about eigenvalue decomposition.” Caption text: “Next, we will discuss the properties of determinants.” The preliminary semantic fusion result (unrevised) is: = “Next, we will discuss the properties of determinants.” Calculate the context similarity: The system determines that there is a semantic mutation, and extracts the previous segment theme word: = {matrix, linear, transformation} Since there is no related term in the current revised result, the system increases the weight of the text related to the previous semantic theme, and recalculates: According to the new weight, the fusion text is regenerated, and the revised output is: “Before talking about eigenvalues, let's review the linear transformation properties of matrices.” The weighted fusion module is preferably used for local correction of semantic conflicts within a single segment, while the context consistency correction module is further used for cross-segment semantic continuity control to prevent theme jumping or logical breakage in the fused text. The two can be executed in sequence, and they are complementary rather than alternative. The following illustrates the specific calculation process of weighted fusion.

[0041] In another preferred embodiment, within a certain time window, the ASR recognized text is: “The determinant of a matrix can be expressed as the sum of algebraic minors.” The corresponding caption text is: “The determinant of a matrix is equal to the sum of all algebraic minors.”

[0042] After performing semantic interpolation, the system generates the revised text: "The determinant of a matrix can be expressed as the sum of each algebraic minor". This result not only preserves the sentence structure of ASR, but also corrects the key term error using subtitle text, thereby significantly improving the recognition quality.

[0043] Through the above structure, the semantic calibration module can automatically balance the reliability of the two data sources in the case of ASR misrecognition or incomplete subtitles, and realize the semantic complementation of voice and subtitles.

[0044] In a further preferred technical solution, the application further introduces a fusion confidence sequence stability analysis mechanism for cross-time backtracking, which is used to control the overall consistency of the fusion text on a longer time scale. Specifically, the system records the fusion weight sequence of the last K fusion segments , which is used to detect whether the fusion confidence is stable. The fusion weight sequence can be: ; wherein, is the fusion weight (weight coefficient) of the current segment. It is used to control the proportion of ASR text and subtitle text in the fusion process. t is the segment time index (segment number) of the current processing, indicating that the system is processing the t-th semantic segment. K is the cross-time backtracking window size, indicating the number of segments analyzed by the system, for example, K = 3, 4, 5. represents the fusion weight of the last K consecutive segments in the time interval . It reflects the trend of the change of the fusion strategy over time.

[0045] In addition, the stability score of the fusion weight sequence needs to be calculated, for example: ; wherein represents the sequence variance, which is used to measure the volatility of the fusion weight; reflects the volatility of a between consecutive segments, which is used to measure whether the confidence is unstable.

[0046] is the context similarity sequence of the corresponding segment; represents the average value of the context similarity sequence, which is used to reflect the overall semantic continuity. The lower the context consistency, the greater this item, and the more significant the impact on the stability score.

[0047] This parameter is used to adjust the balance between confidence stability and contextual consistency. It determines the contribution of contextual similarity to the stability score. Its value can range from 0.1 to 1.0.

[0048] when Exceeding the set threshold ( This is the stability threshold. It's used to determine if there are abnormal fluctuations in the fusion weights (it can be between 0.2 and 0.5). When this threshold is reached, the system determines that there is persistent semantic inconsistency or short-term error accumulation, triggering a backtracking correction mechanism. (System selection before...) A fragment The fusion weight α is recalculated, and the fusion is performed again using the latest contextual features, topic keywords, and semantic similarity matrix. The corrected fused text segment sequence is denoted as: in This represents the fused text after backtracking and correction. N is the number of segments that were backtracked and corrected. ; This represents a new text sequence generated by re-merging the N most recent segments. The backtracking correction involves... The alternative is: for the segment that is backtracked and covered, Used to replace the original For segments that were not backtracked, the following method is still used. .

[0049] This backtracking correction mechanism can significantly reduce the accumulation of segment-level errors caused by temporary noise, speech rate changes, subtitle OCR errors, or ASR misidentification, thereby improving the stability of the entire text sequence and global semantic consistency.

[0050] In a typical embodiment, when an α fluctuation increase exceeding 0.35 is detected in the past four fused segments and at least two segments have contextual similarity... When the error occurs, the system automatically triggers a backtracking correction, recalculating the fusion weights and semantic interpolation for the first three segments, thereby preventing the error from spreading continuously in the topic switching area.

[0051] To ensure the verifiability and determinism of the fusion process between the speech recognition text and the subtitle text, this invention further establishes the following character-level correspondence rules and error constraint mechanisms to enable an interpretable correspondence between the ASR text and the subtitle text: Character Quantity Alignment Rule: Before performing segment-level fusion, the present invention performs character length standardization on the ASR text segment and the subtitle text segment respectively, making the character quantities of the two types of texts consistent, and aligning them word by word at the corresponding positions. When necessary, character-level one-to-one correspondence is achieved through methods such as character padding, truncation, or splitting according to semantic boundaries. This rule enables subsequent weight interpolation, text replacement, and semantic calibration to have clear corresponding relationships, avoiding unverifiable fusion results.

[0052] ASR Error Phonetic Constraint: For the speech recognition scenario, the present invention limits the errors of the ASR output characters to a range with phonetic similarity. Specifically, when the ASR character is inconsistent with the corresponding subtitle character, at least partial consistency in the initial consonant or final must be ensured, such as similar initial consonants (e.g., "j / zh", "c / ch"), similar finals (e.g., "an / ang", "ing / in"), or similar tones. This phonetic similarity constraint makes the ASR errors predictable, enabling the fusion module to more reliably determine whether the ASR characters can be used as credible sources to participate in adaptive weighted fusion.

[0053] Subtitle OCR Error Glyph Constraint: For the subtitle text obtained by OCR recognition, the present invention requires that the incorrect characters and the corresponding target characters maintain partial similarity in visual structure, such as sharing radicals, components, or main stroke structures (e.g., "淀 / 淇", "未 / 末", "阵 / 陈"). This glyph similarity constraint conforms to the typical characteristics of OCR errors, enabling the system to identify and correct visual errors based on character structure rules, and improving the accuracy of subtitle text credibility assessment.

[0054] Through the above character alignment rules and error constraint mechanisms, the semantic calibration process of the present invention does not completely rely on the black-box generation of deep learning models, but establishes an interpretable and repeatable verification corresponding relationship at the character level. Even when sacrificing the smoothness of the language when necessary, the present invention still prioritizes ensuring the certainty at the character level, thereby making the text fusion, semantic interpolation, and context correction processes achievable and technically feasible.

[0055] Semantic Clustering Module 40: Used to perform embedding and similarity calculation on the text segments after semantic calibration, and cluster the content with similar semantics into topic classes. In a preferred embodiment, the following semantic segment splitting algorithm is adopted: Initial splitting (syntactic splitting): Split into basic sentence segments according to Chinese punctuation marks (period, question mark, comma). Semantic adjacency calculation: Calculate the cosine similarity Sim(T i , T i+1 for adjacent sentence segments T i , T i+1Threshold decision merging: if Sim ≥ 0.65, consider the semantics continuous, merge the segments. Length constraint: single segment length no more than 80 words, if exceed, split the sentence boundary by 60 / 40.

[0056] The module receives high-precision text T_final output from the semantic calibration module 30, and converts each text segment into a semantic vector V_i through a semantic embedding model (such as Transformer Encoder or Sentence-BERT). The system calculates the cosine similarity between any two segments to construct a similarity matrix , and uses clustering algorithms (such as K-Means or hierarchical clustering) to group the texts. Each clustering result corresponds to a semantic topic class C_k.

[0057] In a preferred embodiment, the elements of the similarity matrix M(i,j) are calculated by the cosine similarity between the semantic vectors of the text segments, which is defined as: ; where V i and V j represent the semantic embedding vectors of the i-th and j-th text segments, respectively; The value range is [0, 1], and the closer to 1 indicates the more similar semantics.

[0058] The system constructs a matrix M with a size of N x N, where N is the number of text segments; when (preferred threshold δ = 0.75), it is considered that segment i and segment j belong to the same semantic topic candidate set, and are used as input features for the clustering algorithm.

[0059] For example, when the text is divided into 4 segments, the constructed similarity matrix can be shown as follows: According to the matrix, the system automatically identifies high-similarity segment groups, for example , segment 1 and segment 2 are classified into the same topic class; similarly, segment 3 and segment 4 are clustered into another topic class.

[0060] Through the above clustering process, the system automatically merges text segments with similar semantics into several semantic topic classes, each topic class corresponds to a semantic cluster, denoted as C k (k = 1, 2, …, n), where n is the number of topics obtained after clustering. The semantic clustering module 40 finally outputs the topic class set {C1, C2, …, Cn}, and uses this set as the input of the subsequent knowledge unit construction module 50 for performing semantic compression, keyword extraction, and structured knowledge unit generation.

[0061] The module outputs a set of topic classes {C1, C2, …, Cn} and passes them to the knowledge unit construction module 50.

[0062] Knowledge unit construction module 50: used to semantically compress and structure the clustering results, generating standardized knowledge units. This module receives the set of topic classes {C1, C2, …, Cn} from the semantic clustering module 40, and performs semantic summarization and keyword extraction operations on each topic class.

[0063] Specifically, the system extracts core sentences and high-weight keywords from each topic class based on the TextRank summarization algorithm, generating compressed text. Then, the resulting data is encapsulated into a structured knowledge unit in a unified format, preferably represented in JSON format: { "id": "unit_0001", "topic": "Matrix multiplication definition and application", "summary": "Matrix multiplication realizes linear mapping combination through row-column inner product, and is used for solving linear equations and image transformation.", "keywords": ["matrix", "linear mapping", "inner product"] } The final output knowledge unit set Stored in the knowledge base database (which can be a vector database or a relational database) and used as the input data source for the knowledge calling module 60.

[0064] Knowledge calling module 60: used to realize knowledge retrieval, question answering and recommendation based on semantic similarity. This module receives the knowledge unit set {U1, U2, …, Um} from the knowledge unit construction module 50 and stores it in the knowledge database in vector form (e.g. FAISS + PostgreSQL structure). When performing retrieval, the system receives the user's natural language query Q, encodes it into a query vector V_Q through the same semantic embedding model, and calculates the similarity between V_Q and all knowledge unit vectors : The system selects the top several knowledge units with the highest similarity as candidate results and returns the most relevant answers or topic content. Retrieval results Further output to external interfaces or intelligent question answering engines to realize semantic-level knowledge query and recommendation of video content.

[0065] where for calculating the similarity between V_Q and all knowledge unit vectors The semantic similarity is calculated by cosine similarity in a preferred embodiment, which can be defined as: wherein, is the semantic vector of the user query statement; is the semantic vector representation of the knowledge unit . The value range of is [0, 1], and the closer to 1 indicates the higher semantic relevance.

[0066] The system calculates the similarity of all m knowledge units to obtain a similarity sequence: And according to the similarity value, Top-K sorting and filtering are performed, and the K knowledge units with the highest similarity are preferably selected as the return result. Typically, K = 3 or K = 5 can be taken for question answering or recommendation scenarios.

[0067] For example, when the user inputs the query Q = “what is matrix multiplication?”, the system calculates: At this time, the knowledge unit U7 with the highest similarity is output, and its structured content is returned as the response result.

[0068] Embodiment two: A general flow embodiment of the method of the application includes the following stages: Stage S1: Audio and video separation. Extract the audio stream and subtitle stream from the input video. Output the audio file A(t) and the subtitle file S(t).

[0069] Stage S2: ASR recognition. Convert A(t) to text T_ASR using a multi-lingual speech recognition model.

[0070] Stage S3: Semantic calibration. Correct the ASR error by timestamp matching and semantic similarity function to generate the final text T_final.

[0071] In a preferred embodiment, the similarity function is defined as the cosine similarity of semantic embedding vectors.

[0072] wherein, E() is the semantic embedding vector of the text; represents the i-th speech recognition text segment output by the ASR recognition module 20; represents the i-th subtitle text segment obtained by parsing the subtitle file S(t) output by the audio and video preprocessing module 10 according to the timestamp. When (for example, a typical threshold = 0.95), it is determined as a conflict segment, and a context consistency correction strategy is adopted.

[0073] For the i-th , further can be expressed as the i-th subtitle segment obtained by cutting the subtitle text on the time axis according to the frame timestamp. Specifically, the system parses the input subtitle file S(t) (for example, SRT or VTT format), extracts the subtitle time interval [t start ,t end ], and aligns and cuts according to the corresponding time window of the ASR text segment to form a one-to-one corresponding candidate matching segment of the subtitle text and the speech text.

[0074] For example, when the time range of a certain speech segment is 12.5s-16.8s, the system retrieves the subtitle entries falling within the time interval in the subtitle file, and takes the content as ; if the subtitles span multiple entries, they are automatically spliced into the same segment.

[0075] Through the above timestamp alignment process, the system can obtain the speech recognition text 𝑖 corresponding to the subtitle text i in the same time segment. When it is determined that there is semantic inconsistency or recognition error in the segment, the system further performs text fusion operation to generate a final text segment 。

[0076] In a preferred embodiment, the specific fusion calculation formula can be expressed as: ; .

[0077] wherein a is an adaptive weight, for example, a takes 0.8 when Sim is high, and a takes 0.3 when Sim is low, to realize adaptive fusion.

[0078] In a preferred embodiment: the adaptive weight function f(Sim) dynamically adjusts the weight ratio of the ASR and the subtitle text in the fusion process according to the numerical value of the semantic similarity. Its mathematical expression can be defined as: wherein Sim is the semantic similarity (value range [0, 1]); a is the fusion weight coefficient (value range [0, 1]); β0∈[0.1, 0.5], β1∈[0.3, 0.7] are empirical coefficients, which are automatically optimized by a verification set to control the linear response amplitude.

[0079] In a typical embodiment, it is desirable to have β0= 0.3, β1 = 0.5, i.e. α = 0.3 + 0.5 x Sim; When the similarity is high (e.g. Sim = 0.9), α = 0.75, which means the ASR text is more likely to be kept; when the similarity is low (e.g. Sim = 0.4), α = 0.5, which means the caption text is more likely to be kept.

[0080] To enhance the non-linear response effect, in another embodiment, the function f(Sim) can also be defined in sigmoid form as: where k is the steepness coefficient, used to control the transition speed (e.g. k = 8); θ0is the equilibrium point (e.g. θ0= 0.8). The function shows rapid change when Sim is close to 0.8, which can more sensitively adjust the weight, so that the system can adaptively switch the trust source in the semantic boundary segment.

[0081] Stage S4: Semantic clustering and compression. T_final is cut into segments according to the semantic boundary, and a similarity matrix M is calculated. For example, when M

[0082] In a preferred embodiment, the system first cuts the semantically calibrated text T_final into N semantic segments according to the semantic boundary, and encodes each segment into a semantic vector using a semantic embedding model (e.g. Sentence-BERT). Subsequently, the system calculates the cosine similarity between any two segments to construct a similarity matrix M, whose elements are defined as: where The larger the value, the closer the semantics between the segments.

[0083] To realize topic clustering, the system sets a semantic merging threshold δ on the matrix M (preferably δ = 0.75). When M ij ≥ δ, it is considered that the segments S i and S j belong to the same semantic topic candidate set, and both are added to the same topic cluster C k . This process can be iteratively extended, so that each topic cluster is composed of several high-similarity semantic segments: ; When all the segments have completed the attribution, the system outputs the topic class set ; In one example, if It is divided into 4 segments, and the matrix M is as follows: Based on the threshold δ=0.75, two topic clustering results can be obtained: , The system then performs semantic summary extraction on each topic cluster, generates corresponding core knowledge points, and passes them to the knowledge unit construction module 50.

[0084] Phase S5: Knowledge Unit Construction and Storage. The clustering results are transformed into a standardized structure, such as JSON format. JSON { "id": "unit_0001", "topic": "Convolutional Neural Networks", "content": "Explains the feature extraction mechanism of CNN in image recognition." Keyword categories: ["Convolution", "Feature Extraction", "Neural Network"] } Subsequently, the knowledge units are stored in a vector database, supporting similarity retrieval.

[0085] Example 3: This embodiment uses a university-level MOOC video course on linear algebra as an example to illustrate the specific application of the present invention in the field of education.

[0086] The system operating environment can be configured as follows: CPU: Intel Xeon 32-core; GPU: NVIDIA A100; Memory: 256GB. Software environment: ASR model: Whisper-Large; Embedding model: BERT-base-Chinese; Vector database: FAISS+PostgreSQL; Clustering algorithm: Cosine similarity + K-means.

[0087] The specific method steps and calculation process of this embodiment are as follows: Step S1: Audio and video data input and preprocessing. The audio and video processing module 10 downloads the video from the MOOC platform and extracts the audio stream (e.g., a 16kHz sampled WAV file) and the subtitle stream (e.g., an SRT file).

[0088] Step S2: ASR recognition. The ASR recognition module 20 performs speech recognition using the Whisper model. For example, for the audio content: "The determinant of a matrix can be expressed as the sum of algebraic cofactors", the ASR may incorrectly recognize as: "The determinant of a matrix can be expressed as the sum of algebraic cofactors three". In this test, the initial ASR accuracy can reach about 89.2%.

[0089] Step S3: Semantic calibration. The semantic calibration module 30 performs timestamp alignment and semantic fusion on T_ASR and S(t). In a preferred embodiment, a sliding window matching algorithm can be used for time alignment, for example, calculating the time difference When D(i,j) < 0.8s, the segments are considered synchronized. For synchronized segments, the cosine similarity is calculated using the above method. For example, the incorrect text of the ASR and the original text of the subtitles "The determinant of a matrix is equal to the sum of each algebraic cofactor."

[0090] In this embodiment, the system performs semantic encoding on the ASR text "The determinant of a matrix can be expressed as the sum of algebraic cofactors three" and the subtitle text "The determinant of a matrix is equal to the sum of each algebraic cofactor" respectively, obtaining 768-dimensional vector representations: Calculate the cosine similarity: Since (threshold), the system determines that the semantics are consistent, and the more optimal semantic subtitle result is retained. If (another threshold example), the ASR result is replaced with the subtitle text. Through this calibration mechanism, the text accuracy is improved from 89.2% to 95.4%. In a preferred embodiment, to further improve the robustness of the semantic calibration stage in long sequence videos, the present application also performs time sequence monitoring and backtracking correction of fusion weights in stage S3. After completing the weighted fusion of each segment, the system writes the obtained a value into the time sequence, and based on a sliding window model, the system calculates a stability index in real time. When detecting that the fusion weight has a large fluctuation, a sharp decline, or is contrary to the semantic prediction trend within the window, the system triggers the backtracking mode: re-performs semantic embedding, context alignment, and weight allocation for the segments within the window, thereby eliminating the "short-term error accumulation phenomenon". This mechanism is particularly suitable for scenarios where the speaker's accent suddenly changes, the noise suddenly increases, or the subtitles are missing, and can significantly reduce the risk of semantic drift, improve the readability and consistency of continuous segments.

[0091]

[0092] ​​For the stability analysis of the fusion confidence level for performing cross-time backtracking, please refer to Example 1.

[0093] Step S4: Semantic Clustering. The semantic clustering module 40 performs semantic segmentation and clustering on the corrected text. For example, the text is divided into 50 segments. The cosine similarity matrix between the segments is calculated. .when When the value is greater than 0.75, the data are clustered into the same topic cluster. In this embodiment, 50 fragments are clustered into 17 topic classes, such as: C1 (matrix operations), C2 (solving linear equations), C3 (eigenvalues ​​and eigenvectors), C4 (linear transformations), etc. The text compression rate reaches 66%.

[0094] Step S5: Knowledge Unit Generation. The knowledge unit building module 50 performs semantic compression and summary generation for each topic cluster. For example, it uses the TextRank algorithm to extract key sentences, such as: "Matrix multiplication is an algebraic operation defined by the dot product of rows and columns, and is a combination of linear mappings." Structured knowledge units are generated, as shown in the following example: { "topic": "Definition and Application of Matrix Multiplication", "summary": "Matrix multiplication achieves linear mapping combinations through row and column inner products, and is used for solving linear equation systems and image transformations." Keyword: ["matrix", "inner product", "linear mapping"] } Step S6: Knowledge Retrieval and Question Answering. The knowledge retrieval module 60 vectorizes the generated knowledge units and stores them in the FAISS database. When a student inputs the question, "What is matrix multiplication?", the system calculates the query vector. and with all knowledge unit vectors in the knowledge base Calculate correlation The calculation process also uses the aforementioned semantic similarity—cosine similarity—for calculation.

[0095] The system calculates all knowledge units. Then, sort them by similarity from high to low, and take the Top-K results as the candidate knowledge unit set. When a user enters the query Q="What is matrix multiplication?", the system encodes the following: The vector for a knowledge unit "Definition and Application of Matrix Multiplication" in the knowledge base is: but: If the value is the highest among all knowledge units, the system returns as the answer.

[0096] After the vector similarity calculation is completed and the result is returned, the system can further statistically analyze the overall retrieval effect to verify the effectiveness of the method of the application.

[0097] The search result shows that the similarity of the "matrix multiplication definition and application" unit is the highest (for example, 0.94), and is hit and returned. The search accuracy of the method of the application reaches 92% (the traditional keyword matching is 68%), and the average response time is 6s (the traditional one is 15s).

[0098] The beneficial effects of the application are as follows: In the professional term and noise scene, the accuracy and robustness can be improved. Through the semantic calibration fusion of ASR and subtitles, the text accuracy is significantly improved. In addition, the knowledge is structured. Through semantic clustering and knowledge unit construction, the long video text is converted into structured knowledge points, which is convenient for storage and reuse. Based on the vector storage and semantic similarity matching, efficient semantic retrieval and intelligent question answering are realized. In addition, the knowledge compression rate is high (for example, 66%), which significantly improves the reuse efficiency of video knowledge.

[0099] The application realizes high-precision conversion from video to knowledge by introducing the "ASR and subtitle semantic calibration mechanism" and the "semantic clustering knowledge unit method", and significantly improves the speech recognition accuracy, knowledge aggregation efficiency and semantic retrieval performance.

[0100] Those skilled in the art can understand that the drawings are only schematic diagrams of an embodiment, and the modules or flows in the drawings are not necessarily necessary for implementing the application.

[0101] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the embodiment description, or can be correspondingly changed and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further split into multiple sub-modules.

[0102] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. A method for constructing a knowledge base oriented towards video content, characterized in that, The method includes the following steps: S1. Obtain the input video and extract the audio stream and subtitle text; S2. Use a speech recognition model to recognize the audio stream and generate the first text sequence; S3. Based on the timestamp alignment strategy, the first text sequence is matched with the subtitle text, and a semantic embedding model is further used to vectorize the matched segments respectively, and the segment-level semantic similarity is calculated. S4. Based on the aforementioned similarity, dynamically fuse the multi-source texts and output a second text sequence; S5. Cluster the semantic segments of the second text sequence to obtain multiple semantic topic clusters; S6. Perform feature extraction on each semantic topic cluster to generate structured knowledge units; S7. Vectorize the knowledge units and store them in a vector database for subsequent retrieval.

2. The method according to claim 1, characterized in that, In step S3, the timestamp alignment strategy includes: When one ASR segment corresponds to multiple subtitle segments, the multiple subtitle segments are concatenated and used as the matching object; when multiple ASR segments correspond to one subtitle segment, the multiple ASR segments are merged and used as the matching object.

3. The method according to claim 1, characterized in that, In step S4, the dynamic fusion of multi-source texts based on the similarity and the output of the second text sequence includes: calculating the fusion weight α based on semantic similarity Sim, where... Wherein, β0 and β1 are preset coefficients, and α is used to perform weighted interpolation on the first text sequence and the subtitle text to generate the second text sequence.

4. The method according to claim 1, characterized in that, Step S4 also includes context consistency correction, specifically including: When the contextual similarity between the current fused segment and the previous fused segment is lower than a set threshold, the fusion weight α is dynamically adjusted based on the topic keywords of the historical segments, and the calibration text is regenerated.

5. The method according to claim 1, characterized in that, In step S5, the clustering of semantic segments of the second text sequence includes: semantic segmentation of the semantic segments of the second text sequence, initial segmentation based on punctuation, and semantic similarity calculation of adjacent segments. When the similarity is greater than a preset value, the segments are merged, and the length of a single segment does not exceed a preset number of characters.

6. A method for cross-temporal backtracking stability correction for multi-source video-text fusion, characterized in that, include: S1. Obtain the multi-source text corresponding to the video segment, including speech recognition text and subtitle text, and calculate the fusion weight based on the semantic features of the two. To obtain the current fused text fragment ; S2. The fusion weights are... Write the fusion weight sequence arranged in chronological order And extract the fusion weights of the K most recent segments within a preset time window K; ; S3. Based on the fusion weight sequence Calculate stability score Stability rating includes: ;in, Represents the series variance. Let λ be the context similarity sequence of the corresponding segment, and λ be the stability adjustment coefficient. S4. Calculate the stability score. With preset threshold When comparing, When this happens, a backtracking correction operation is triggered, and the most recent N fused segments are re-fused and recalculated. S5. Recalculate the fusion weights of the most recent N fusion segments based on the updated semantic features, contextual consistency, and historical topic keywords. And generate a backtracked and corrected fused text sequence. To improve the temporal stability and global semantic consistency of text fusion; wherein the backtracking correction .

7. A knowledge base construction system for video content, characterized in that, The system includes: an audio and video preprocessing module, a speech recognition module, a semantic calibration module, a semantic clustering module, a knowledge unit construction module, and a knowledge base calling module; The audio and video preprocessing module is used to extract the audio stream and subtitle text; The speech recognition module is used to generate a first text sequence based on the audio stream; The semantic calibration module is used to perform timestamp alignment, semantic similarity calculation, adaptive fusion, and language model generation, and outputs the calibrated second text sequence. The semantic clustering module is used to cluster the semantic segments of the second text sequence to obtain semantic topic clusters; The knowledge unit construction module is used to generate structured knowledge units based on the semantic topic clusters, and to vectorize and store the knowledge units in a vector database. The knowledge base invocation module, based on the vector database, is used for subsequent retrieval and invocation.

8. The system according to claim 7, characterized in that, The system further includes a semantic segmentation module; the semantic segmentation module is used to segment the second text sequence according to semantic boundaries to obtain semantic segments of the second text sequence.

9. The system according to claim 7, characterized in that, The semantic calibration module includes: The system includes a time alignment unit, a similarity calculation unit, and a weighted fusion unit.

10. The system according to claim 7, characterized in that, The semantic calibration module further includes a context consistency correction unit. When the context similarity between the current fused segment and the previous fused segment is lower than a set threshold, the context consistency correction unit dynamically adjusts the fusion weight α according to the topic keywords of the historical segments and regenerates the calibration text.

Images