Method and system for automatically constructing multi-modal knowledge graph from heterogeneous video data sources

Abstract

The present application relates to a kind of from isomeric video data source automatically constructs multimodal knowledge graph method and system, wherein, method includes: S0, access eight categories of source such as caption, dialogue, poster, still, music metadata, cast, plot synopsis, external score link;S1, according to preset cross section unit division and extraction candidate triple;S2, according to entity equivalence matching rule merge homologous equivalence appearance;S3, the stability component is calculated to each triple in the appearance sequence of each source, component takes the skewness of normalized cross section unit interval distribution and transforms;S4, splice into multi-source consistency vector;S5, according to two-level threshold and two-level quantity threshold determination core, routine, edge three categories, according to corresponding fusion weight write in atlas.This method does not introduce additional training data and big model, single point calculation complexity is constant, so that core triple and edge corner guest string confidence form clear distinction.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph construction technology, specifically to a method and system for automatically constructing a multimodal knowledge graph from heterogeneous film and television data sources. Background Technology

[0002] Knowledge graphs in the film and television industry play a crucial role in downstream applications such as character retrieval, plot-based question answering, and content recommendation. Early film and television knowledge graphs relied heavily on manual annotation, with editorial teams inputting structured fields such as actors, characters, and plot details for each work. This approach was costly, had limited coverage, and could not keep up with the demands of large-scale processing.

[0003] To address this, the industry has proposed various methods for automatically constructing film and television knowledge graphs from heterogeneous data sources. Mainstream approaches include: using large language models to extract entities and relationships from texts such as subtitles and synopses; using visual language models to identify faces and scenes from images such as posters and stills; and using multimodal fusion graph neural networks to align and fuse candidate triples from different sources. These methods show significant improvements in coverage and recall compared to manual annotation. Building on this, the industry has proposed various candidate triple scoring strategies based on a hybrid of rules and statistics, such as multi-source fusion strategies weighted by cross-source frequency of occurrence and strategies for discrete scoring based on cross-source type groups. While these methods do differentiate the credibility of candidate triples to some extent without relying on additional training data, their scoring criteria still primarily focus on two dimensions: "whether it appears in a certain source" and "the total number of occurrences," without providing detailed modeling of the consistency of occurrence across different time segments within the same source.

[0004] In practical engineering, this approach reveals a common problem. Take a typical TV series as an example: a minor guest character appears only once in the end credits and also once in the final cast list, resulting in a cross-source sampling category 2; while a core character throughout the series appears in multiple forms—name, face, and scene relationships—in credits, dialogue, posters, stills, and the cast list, resulting in a cross-source sampling category 6 to 8. When judging solely by the total number of cross-source appearances, the former receives a high score due to being sampled across all sources, thus ranking high in downstream recommendations; the latter, however, suffers from diluted confidence due to the sheer volume of appearances. Upon closer examination, the root cause lies in the fact that existing methods only consider whether candidate triples appear in the cross-source spatial dimension, failing to utilize the evenness of their appearance across different time segments within the same source. True core characters often appear evenly and repeatedly, while minor guest characters are concentrated in one or two time points.

[0005] Therefore, the following technical problems need to be solved when automatically constructing a film and television knowledge graph: First, how to identify the cross-segment consistency of each candidate triple within each type of heterogeneous source without introducing additional training data or relying on fine-tuning of domain knowledge of a large model; Second, how to integrate the consistency information of multiple sources into a unified weighting basis, so as to distinguish core, regular and marginal triples according to semantic importance when writing into the graph; Third, how to make the mechanism lightweight, interpretable and easy to engineer, so that it can run in real time in a large-scale film source processing pipeline. Summary of the Invention

[0006] Through multiple experiments, it was found that by using the statistical pattern of the cross-segment unit interval distribution of each candidate triple within the same type of heterogeneous source as the weighting criterion, a clear distinction can be made between the confidence levels of core members and peripheral members without introducing any additional training data or calling any large models. Moreover, this statistical pattern is expressed as a single scalar, which is easy to deploy in engineering and for later auditing. Based on this finding, the specific technical solution of this invention is as follows.

[0007] The technical problem to be solved by this invention is to provide a method and system for automatically constructing a multimodal knowledge graph from heterogeneous film and television data sources, so as to solve the problem that the existing automatic construction route only makes binary or frequency judgments on candidate triples based on whether they appear in the spatial dimension, and fails to utilize the consistency of the time dimension across segments within the same source, resulting in misalignment of the confidence of core and corner triples.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution.

[0009] In a first aspect, the present invention provides a method for automatically constructing a multimodal knowledge graph from heterogeneous film and television data sources, comprising the following steps: S0, access eight types of heterogeneous film and television data sources, namely, subtitle source, dialogue source, poster source, still source, background music metadata source, cast list source, plot synopsis source, and external rating link source; S1, for each type of heterogeneous video data source, divide it according to the preset cross-segment unit, and extract candidate triples from each cross-segment unit; S2, for each candidate triplet, according to the preset entity equivalence matching rule, merge the equivalent occurrences of different cross-segment units in the same type of heterogeneous film and television data source to obtain the cross-segment unit occurrence sequence of the candidate triplet in the source of the same type. S3, For each candidate triplet in the sequence of occurrence of cross-segment units in each source class, calculate the stability component. The stability component is obtained by taking the absolute value of the skewness value of the normalized cross-segment unit interval distribution and adding a preset constant inversely. S4. For each candidate triplet, its stability components on the eight sources are concatenated into a multi-source consistency vector in a preset order. S5. Based on the multi-source consistency vector of each candidate triplet, each candidate triplet is divided into one of the following: core triplet, regular triplet, or edge triplet. The specific steps are as follows: First, count the number of components in the multi-source consistency vector of each candidate triplet whose component value is not less than a preset first threshold. If the number of components is not less than the preset first threshold, then the candidate triplet is classified as a core triplet. Second, count the number of components in the multi-source consistency vector of each candidate triplet whose component value is less than the preset first threshold and not less than a preset second threshold. If the number of components is not less than the preset second threshold, then the candidate triplet is classified as a regular triplet. Other candidate triplets besides the core triplet and regular triplet are classified as edge triplets. Candidate triples are written into the multimodal knowledge graph according to their corresponding fusion weights. The corresponding fusion weight of core triples is greater than that of regular triples, and the corresponding fusion weight of regular triples is greater than that of edge triples.

[0010] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method.

[0011] Thirdly, this invention provides a system for automatically constructing a multimodal knowledge graph from heterogeneous film and television data sources. The system comprises six interconnected modules: a data source access module, a candidate triplet extraction module, a cross-segment equivalence merging module, a stability component calculation module, a multi-source consistency vector splicing module, and a classification and graph writing module. These modules are used to execute steps S0 to S5 of the aforementioned method. As shown in Figure 2, the six modules are connected sequentially. The output of the data source access module is connected to the input of the candidate triplet extraction module, the output of the candidate triplet extraction module is connected to the input of the cross-segment equivalence merging module, and so on, until the classification and graph writing module writes the triplet and its corresponding fusion weight into the multimodal knowledge graph stored externally.

[0012] The benefits of this invention are mainly in three aspects. First, by modeling the skewness of the distribution of each candidate triplet across different segments within the same source class, the continuity in the time dimension is incorporated into the weighting of candidate triples, making it possible to distinguish the confidence of core triples from corner triples, thereby reducing the downstream sorting misalignment rate. Second, each candidate triplet is stored using only one scalar stability component for each source class, with constant single-point computational complexity. It does not rely on additional training data or large domain models and can run in real-time in large-scale source processing pipelines. Third, the four preset parameters—first threshold, second threshold, first quantity threshold, and second quantity threshold—are all readable scalars, allowing maintenance personnel to adjust parameters according to data distribution without modifying the model, facilitating auditing. Attached Figure Description

[0013] Figure 1 is a schematic diagram of the overall process of the method described in this invention.

[0014] Figure 2 is a schematic diagram of the connection relationship between the six modules of the system described in this invention.

[0015] Figure 3 is a schematic diagram of the calculation process of the stability component in step S3 of the present invention.

[0016] Figure 4 is a schematic diagram of the values ​​of the eight components of the multi-source consistency vector in step S4 of the present invention.

[0017] Figure 5 is a schematic diagram of the three-class classification determination of candidate triplet in step S5 of the present invention.

[0018] Figure 6 is a schematic diagram comparing the performance of the embodiment of the present invention and the baseline method in terms of downstream sorting misalignment rate. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention. The specific threshold values, quantity thresholds, fusion weights, default values, facial similarity thresholds, normalization constants, etc., used in the following embodiments are all illustrative examples and do not constitute a limitation on the scope of protection of this invention; those skilled in the art can adjust the corresponding parameters according to actual application scenarios, and the adjusted implementation methods still fall within the scope of protection of this invention. The following specific embodiments are presented in the following order: First, the overall flow of the method described in this invention is summarized with reference to Figure 1, and the connection relationship of the six modules of the system is introduced with reference to Figure 2; second, the core terms and the underlying steps of the core algorithm involved in this invention are explained in detail; then, three specific embodiments are presented in sequence; finally, experimental comparisons with existing baselines, boundary conditions, and anomaly handling are given.

[0020] The core terms involved in this invention will be explained below.

[0021] The heterogeneous film and television data sources refer to data sources related to the same film or television work, obtained using different data formats and acquisition methods, and provided as input to this method. This invention limits them to eight categories, including subtitle sources, dialogue sources, poster sources, still images, background music metadata sources, cast list sources, synopsis sources, and external rating link sources. Specifically, the subtitle source is a subtitle text stream aligned to the timeline; the dialogue source is a dialogue text stream obtained through speech-to-text transcription or script alignment; the poster source is a collection of poster images related to the work; the still image source is a collection of still images related to the work; the background music metadata source is a structured metadata collection of the work's background music tracks; the cast list source is structured data of the work's actor list; the synopsis source is a short text synopsis of the work's plot; and the external rating link source is a collection of links pointing to external rating pages for the work.

[0022] The term "segment unit" refers to the smallest segment used to divide the content of each type of heterogeneous film and television data source and define the sequence of occurrence of triples within that source. The definition of segment units differs for different sources. Typically, for subtitle sources, the segment unit is the subtitle segment obtained by dividing the film source into scenes; for dialogue sources, the segment unit is the dialogue segment obtained by dividing dialogue characters' speech; for poster sources, the segment unit is each poster image; for still image sources, the segment unit is each still image; for music metadata sources, the segment unit is each piece of music; for cast list sources, the segment unit is the cast list entry for each actor; for synopsis sources, the segment unit is the sentence segment obtained by dividing the synopsis into paragraphs; and for external rating link sources, the segment unit is each external rating page entry. Each segment unit is assigned a natural number sequence within that type of source according to its order of appearance, starting from 1 and increasing sequentially.

[0023] The entity equivalence matching rule refers to the set of judgment rules used in step S2 to determine whether entities appearing in different cross-segment units refer to the same real object; the present invention is limited to three rules judged in the above order, namely, name normalization matching is used for personal name entities, face embedding vector cosine similarity matching is used for facial entities, and scene coreference matching is used for other cases.

[0024] The stability component refers to the sequence of occurrences of each candidate triplet across segments in a certain source class. First, the segment interval sequence is obtained by the difference between the indices of two consecutive occurrences of the segment unit. Then, it is normalized to the total number of segment units corresponding to that source to obtain a normalized interval sequence. The sample skewness is calculated on the normalized interval sequence, and its absolute value is inverted and then added to a preset constant to obtain a scalar. The preset constant is typically 1.0. The stability component ranges from 0 to 1. When the value exceeds this closed interval, it is truncated back to the closed interval (less than 0 is truncated to 0, greater than 1 is truncated to 1). The closer the value is to 1, the more uniform the normalized interval distribution, reflecting the more stable the triplet appears in that source; conversely, the more skewed it is, the less stable it appears.

[0025] The multi-source consistency vector refers to an eight-dimensional real-number vector obtained by concatenating the eight stability components corresponding to each candidate triplet across eight heterogeneous film and television data sources in a preset order. The preset order specifically includes the stability components of the subtitle source, dialogue source, poster source, still image source, background music metadata source, cast list source, synopsis source, and external rating link source. For candidate triplets not appearing in a certain source category, a preset default value is taken at the corresponding position. This preset default value is typically 0.05 and is lower than a preset second threshold value.

[0026] The preset first threshold refers to the threshold for determining whether a single stability component meets the standard in the core file, with a value ranging from 0.6 to 0.9, typically 0.7. The preset second threshold refers to the threshold for determining whether a single stability component meets the standard in the regular file, with a value ranging from 0.3 to 0.6, typically 0.45; both satisfy the condition that the first threshold value is higher than the second threshold value. The preset first quantity threshold refers to the threshold for determining whether the number of compliant components in the core file triggers the threshold for that file, with a value ranging from 5 to 8, typically 6. The preset second quantity threshold refers to the threshold for determining whether the number of compliant components in the regular file triggers the threshold for that file, with a value ranging from 3 to 6, typically 4; both satisfy the condition that the first quantity threshold value is higher than the second quantity threshold value.

[0027] The core triplet refers to the candidate triplet with the highest semantic importance that is consistently appearing in multiple sources and is classified according to the first tier determination in step S5. The regular triplet refers to the candidate triplet with moderate semantic importance that is consistently appearing in a moderate number of sources and is classified according to the second tier determination in step S5. The marginal triplet refers to the candidate triplet with the lowest semantic importance that is classified according to the first tier determination in step S5, does not meet the criteria of the core tier or the regular tier determination, and is consistently appearing only in a few sources or unstable in most sources.

[0028] The corresponding fusion weight is the confidence value attached to the candidate triples when writing them into the multimodal knowledge graph. The corresponding fusion weight of the core triples ranges from 0.6 to 1.0, with a typical value of 0.85; the corresponding fusion weight of the regular triples ranges from 0.3 to 0.6, with a typical value of 0.45; the corresponding fusion weight of the edge triples ranges from 0.05 to 0.3, with a typical value of 0.15; the corresponding fusion weight values ​​of core, regular, and edge triples decrease sequentially.

[0029] The multimodal knowledge graph refers to the target graph constructed and written into candidate triples in this invention. Its nodes include entity nodes and attribute key-value pairs, and its edges include relationships between entities. Each node or edge is assigned a confidence attribute, which is written or updated by the corresponding fusion weights of this invention. When the same node or edge appears repeatedly in different candidate triples, it is sorted in descending order according to the corresponding fusion weight values ​​of the repeated candidate triples, and the first value is taken as the final confidence of the node or edge.

[0030] The facial similarity threshold is the cosine similarity threshold used in step S2, scenario two, to determine whether two facial embedding vectors belong to the same facial entity. It is typically set to 0.65, with a range of 0.55 to 0.85, and can be adjusted according to the facial recognition model used and the average illumination conditions of the source image.

[0031] Example 1 provides an end-to-end general implementation of the method described in this invention, progressively unfolding each step from S0 to S5. As shown in Figure 1, this method is executed sequentially from steps S0 to S5; correspondingly, as shown in Figure 2, this system consists of six interconnected modules: a data source access module, a candidate triplet extraction module, a cross-segment equivalent merging module, a stability component calculation module, a multi-source consistency vector splicing module, and a classification and graph writing module. These six modules sequentially perform steps S0 to S5. Each step is described in detail below.

[0032] Step S0: Access Eight Types of Heterogeneous Film and Television Data Sources. This step establishes a stable access channel from this method to eight heterogeneous sources for the film and television works to be processed. In implementation, this step can be implemented as the entry function of a data source access module, whose parameter is the unique identifier of the film and television work to be processed (e.g., the internal work ID of a streaming media platform or the work number in a public dataset). The module establishes access connections sequentially to the subtitle source, dialogue source, poster source, still image source, background music metadata source, cast list source, synopsis source, and external rating link source according to the preset access interface protocol. Each source corresponds to a preset access function, which returns the access handle, original content, and basic metadata of that source (e.g., the total number of scenes in the subtitle source, the total number of images in the poster source). Adjustable parameters include the access timeout and retry count for each source. To illustrate, S0 is like a library's borrowing procedure: first, the access credentials for each type of resource are processed, and subsequent steps only need to focus on the resource content, without being interrupted by access layer failures.

[0033] Step S00 (optional preprocessing step): Before S0, each type of source is cleaned according to its format specification, including removing duplicate entries, unifying character encoding, decoding binary information in image and audio metadata into structured fields, and performing global mapping preprocessing of alias dictionaries for personal names. S00 can alleviate the sequence splitting caused in step S2 due to the existence of multiple equivalent expressions for the same entity but the equivalence matching rules failing to identify a certain expression; the dictionary size of S00 is typically 50,000 to 500,000 records, and can be appropriately expanded due to regional and temporal differences.

[0034] Step S1: For each type of heterogeneous film and television data source, divide it according to preset cross-segment units, and extract candidate triples from each cross-segment unit. The division of cross-segment units follows the preset rules for each type of source: subtitle sources are divided by film scene (usually 30 to 100 scenes per episode), dialogue sources are divided by character speech (usually 300 to 800 dialogue segments per episode), poster sources are independent for each image (usually 1 to 20 images), still images are independent for each image (usually 50 to 200 images), music metadata sources are independent for each track (usually 10 to 30 tracks), cast list sources are independent for each actor entry (usually 50 to 200 entries), synopsis sources are divided by paragraphs (usually 10 to 30 sentence segments), and external rating link sources are independent for each page entry (usually 3 to 15 entries).

[0035] Candidate triple extraction employs a general <entity, relation, attribute> recognition process: For text-based sources (subtitles, dialogue, plot synopsis), name entity recognition is performed first, followed by scene relation recognition and attribute key-value pair recognition; for image-based sources (posters, stills), face detection is performed first to obtain facial embedding vectors, with the appearance of the face in the image serving as the ontology of the candidate triple; for structured sources (music metadata, cast list, external rating links), candidate triples are directly extracted by field. Each candidate triple is in the form of (subject, predicate, object) or (subject, attribute key, attribute value), and is appended with (source identifier, cross-segment unit number) as metadata. If extraction fails in a cross-segment unit (e.g., a poster has no recognizable face), the corresponding candidate triple set for that cross-segment unit is empty, without affecting the processing of other cross-segment units. The adjustable parameter for this step is the extraction confidence threshold for each source.

[0036] Step S2: For each candidate triple, merge the equivalent occurrences of different cross-segment units in the same source according to the preset entity equivalence matching rules. The entity equivalence matching rules determine three cases in the above order: Case 1, name normalization matching, that is, comparing the strings after removing spaces, unifying simplified and traditional characters, and unifying aliases of the name; Case 2, matching where the cosine similarity of the face embedding vector is not less than the face similarity threshold (usually 0.65); Case 3, scene coreference matching, that is, the relational entities linked by the existing equivalent entities in the same cross-segment unit are classified as equivalent to the entity. Taking the name entity "Zhou XX" as an example, it appears in the 3rd, 17th, and 32nd scenes of the subtitle source in the forms of "Zhou XX", "XX", and "Miss Zhou" respectively; after name normalization matching (the normalized character table completes "XX" to the surname "Zhou" and maps "Miss Zhou" to the main name), the three are identified as the same entity, and after merging, the cross-segment unit occurrence sequence of the entity in the subtitle source is [3, 17, 32]. To use an analogy: S2 is equivalent to aggregating different titles for the same person appearing in different chapters onto a single "character card," providing a clean input for determining temporal continuity. Adjustable parameters include facial similarity thresholds and a normalized character table for names.

[0037] Step S3: For each candidate triplet, calculate the stability component for the sequence of occurrences across segments in each source class. As shown in Figure 3, this step is executed sequentially according to sub-steps S3a, S3b, and S3c.

[0038] S3a, the sequence of candidate triplet occurrences across segments in the source. The interval sequence of the segmented unit is calculated based on the difference between the sequence numbers of two consecutive segmented units. The length of this interval sequence is .

[0039] S3b, for the segmented unit interval sequence According to the total number of cross-segment units corresponding to the source Normalization (e.g., for subtitle sources) (Take the total number of subtitle scenes for the series) to obtain the normalized interval sequence. , Each element in the set has a value between 0 and 1.

[0040] S3c, for normalized interval sequences Calculate sample skewness The standard formula for calculating sample skewness is: ,in For sequence length, The mean of the sequence. is the standard deviation of the sequence samples. For the obtained... Take the absolute value, then invert it and add a preset constant. (Usually taken as 1.0), that is, stability component = When the value exceeds the closed interval of 0 to 1, it is truncated and mapped back to the closed interval. The adjustable parameter is a preset constant. .

[0041] Let's discuss the physical meaning: If a triplet appears once every roughly the same number of frames within the source, then the values ​​of the normalized interval sequence are concentrated around the mean, and the absolute value of the skewness is small. A value close to 1 indicates stability; conversely, if it is concentrated in a certain segment and does not appear in other scenes, then the interval sequence contains both very small values ​​(appearing consecutively) and very large values ​​(not appearing again), resulting in a large absolute value of skewness. A value close to 0 indicates instability. The advantage of this transformation is that the stability component is a single scalar and dimensionless number, comparable across sources, and facilitates database entry and auditing in engineering. Failure scenario simulation: If the candidate triplet appears only once in the source, the interval sequence is empty, and skewness cannot be calculated. In this case, the stability component takes a preset default value (usually set to 0.05). If it appears only twice, the interval sequence length is only 1, and the skewness cannot be defined. In this case, another preset value is taken (usually set to 0.20, still below the second threshold, to avoid false positives for single-point occurrences). If it appears three times or more, it is calculated normally according to the above formula.

[0042] Step S4: For each candidate triplet, concatenate its stability components across the eight source classes into a multi-source consistency vector in a preset order. The purpose of this step is to integrate the stability components of each candidate triplet across the eight source classes into an 8-dimensional vector, which serves as the input for the decision in S5. An example of the data structure is shown below. Each of them The value range is 0 to 1. The splicing order is as follows: stability components corresponding to the subtitle source, dialogue source, poster source, still image source, background music metadata source, cast list source, plot synopsis source, and external rating link source. For example... Figure 4 As shown, The eight components are coded positionally and correspond one-to-one with the source type; for candidate triples that do not appear in a certain type of source, the corresponding position... The default value is set (typically 0.05), which is lower than the second preset threshold value. The adjustable parameter is the default value.

[0043] Step S5: For each candidate triplet, determine one of the three categories based on the relationship between the multi-source consistency vector and the four thresholds, and write it into the multimodal knowledge graph according to the corresponding fusion weight. For example... Figure 5 As shown, this step divides candidate triples into three categories—core, regular, and marginal—along the axis of high-level stable components. The determination is made in the following order: First category, statistical analysis... The mid-value is not less than the preset first threshold. (Usually taken as 0.7) number of components ,like Not less than the preset first quantity threshold (Usually 6) Then the candidate triple is classified as the core triple and assigned the corresponding fusion weight. (Usually taken as 0.85); Second tier, when the core conditions of the first tier are not met, statistics are... The mid-value is not less than the preset second threshold. (Typically taken as 0.45) number of components ,like Not less than the preset second quantity threshold (Usually 4) then the candidate triple is classified as a regular triple and assigned the corresponding fusion weight. (Typically 0.45); other cases are classified as edge triples and assigned corresponding fusion weights. (Usually taken as 0.15).

[0044] Subsequently, the candidate triple is written into the multimodal knowledge graph according to its corresponding fusion weight: a corresponding entity node, relation edge, or attribute key-value pair is created in the graph, and the corresponding fusion weight is written as the confidence attribute of the entity node, relation edge, or attribute key-value pair. For cases where the same entity node or relation edge appears repeatedly in different candidate triples, the first value of the repeated candidate triples is taken as the final confidence of the node or edge after sorting them in descending order of their corresponding fusion weight values. For example, if an entity node of a key figure is involved in multiple core triples, and the corresponding fusion weight written each time is 0.85, then taking the first value in descending order will still result in 0.85; if the same node is later involved in a regular triple with a weight of 0.45, then taking the first value in descending order will still result in 0.85, and the confidence will not be overwritten by a low value. This writing strategy ensures the stable visibility of core entities in the graph.

[0045] Overview of adjustable parameters: , , , Default value, facial similarity threshold, normalization constant There are 7 items in total. Operation and maintenance personnel can adjust each item independently according to the different needs of downstream applications for precise recruitment, such as the source structure, extraction coverage, and the emphasis of downstream applications, without having to retrain any model.

[0046] The system implementation corresponding to this embodiment is as follows: the above six modules are connected in a serial order, and the input of each module is the output of the previous module. Media implementation: the above method can be packaged into an executable program, stored in a computer-readable storage medium, and executed in the order of S0 to S5 during loading and execution.

[0047] Example 2 illustrates the distinction between core characters and supporting roles in a 12-episode urban drama. The drama's subtitle source contains 612 storyboards, 4863 dialogue segments, a cast of 87 actors, 8 posters, 184 stills, 23 background music tracks, 18 synopsis segments, and 6 external rating links.

[0048] The lead actor "Lin X" appears in the subtitles of episode 12 in three forms: "Lin X," "Old Lin," and "General Manager Lin." After name normalization and merging according to the entity equivalence matching rule, the sequence length of appearance in the subtitle source is 213 (i.e., spanning 213 scenes), with a stability component of 0.91; the sequence length of appearance in the dialogue source is 198, with a stability component of 0.88; it appears in 7 out of 8 posters, with a stability component of 0.83; it appears in 168 out of 184 stills, with a stability component of 0.87; it exists as the character's theme song in the background music metadata source, with a stability component of 0.72; it is listed first in the cast list, with a stability component of 1.00; it appears in 14 out of 18 sentence segments in the plot synopsis source, with a stability component of 0.79; and it is mentioned in all 5 external rating links, with a stability component of 0.74.

[0049] The multi-source consistency vector of this role (Lin X) is obtained by concatenating the above sequence. The number of qualified components is counted based on the first threshold of 0.7. The value is no less than the first quantity threshold of 6, so it is classified as a core triple and assigned a corresponding fusion weight of 0.85.

[0050] In contrast, the cameo appearance of "Neighbor Lao Zhang" at the end of episode 11 only appears once in the 587th frame of the subtitle source, and once again in the actor list as "Lao Zhang / played by Sun XX," but not in the other six source categories. The multi-source consistency vector corresponding to Lao Zhang... The actor's name appears only once, so the "appears once" branch uses the preset value of 0.20; the other six categories use the default value of 0.05 if they do not appear. The first threshold of 0.7 is used for judgment. It does not meet the core requirements; it is judged according to the second threshold of 0.45. It does not meet the requirements of the standard set; it is classified as an edge triple and assigned a corresponding fusion weight of 0.15.

[0051] After being written into the multimodal knowledge graph, the confidence score of the character node of the lead actor "Lin X" is 0.85, and the confidence score of the character node of the guest actor "Lao Zhang" is 0.15. The two form a significant priority distinction in downstream recommendation and ranking. The other two characters with mid-stage scenes in the same episode, "Zhou X" and "Aunt Chen", are respectively classified into core triples and regular triples (Aunt Chen's multi-source consistency vector). Number of components meeting the first threshold The number of components that do not meet the standard, but meet the second threshold. (Not less than 4, according to the norm), the confidence levels are 0.85 and 0.45 respectively, which is consistent with the results of manual review.

[0052] Example 3: Distinguishing between routine and peripheral scenarios during large-scale operation in a 100-episode long-running TV series. The series includes 9842 subtitle source scenes, 78315 dialogue segments, a cast of 312 actors, 18 posters, 2841 stills, 87 background music tracks, 78 synopsis segments, and 14 external rating links.

[0053] The second male lead, "Zhou X," appears 1023 times in the subtitle source, with a stability component of 0.81 calculated based on the order of appearance; 894 times in the dialogue source, with a stability component of 0.76; appears in 9 out of 18 posters (mostly posters for later episodes), with a stability component of 0.65; appears in 312 out of 2841 stills (concentrated in episodes 18 to 76), with a stability component of 0.42; exists as the theme song in the background music metadata source, with a stability component of 0.30; is listed second in the cast list, with a stability component of 0.95; appears in 24 out of 78 sentences in the synopsis source, with a stability component of 0.53; and is mentioned in 11 out of 14 external rating links, with a stability component of 0.71.

[0054] The multi-source consistency vector corresponding to Zhou X Determined based on the first threshold of 0.7. (Subtitles, dialogue, cast list, external ratings) do not meet the core requirements; judged according to the second threshold of 0.45. (Adding posters and plot synopsis), not less than 4, classified as a regular triplet, and assigned a corresponding fusion weight of 0.45. This result matches the actual screen time of the second male lead in the drama, preventing the character from being mistakenly promoted to the first male lead by the algorithm (the first male lead in the drama meets the standard component count in 8 source categories). The core file (with a confidence level of 0.85) is located in this series. There are also dozens of guest characters who only appear in episodes 1 to 2. Their multi-source consistency vectors take the default value of 0.05 on multiple sources, and the stability components are generally lower than 0.25, which are classified as marginal triples with a confidence level of 0.15. They will not be mistakenly promoted to the regular file or above when sorted downstream, which is consistent with the understanding of the actual importance of the characters in the series.

[0055] The following provides experimental verification and performance comparison between the implementation method of this invention and existing baseline methods.

[0056] To verify the beneficial effects of this invention, experiments were conducted on the publicly available film and television datasets MM-IMDb and IMDB-Top1000. Three baseline methods were selected: Baseline A is a log-normalized weighted average of cross-source occurrences (corresponding to the cross-source weighted approach), Baseline B is a multi-source fusion weighted average based on an attention mechanism (corresponding to the graph attention network approach), and Baseline C is a cross-modal similarity-based filtering approach (corresponding to the visual-language model fusion approach). Performance metrics included the downstream ranking misalignment rate of the core triple (i.e., the proportion of misaligned entries in the top-100 character recommendations), construction time cost (the total time spent processing 1000 works), and interpretability score (the average score after independent evaluation by three human annotators from 1 to 5).

[0057] On the MM-IMDb dataset, the method of this invention reduces the downstream ranking misalignment rate from 18.4% of baseline A to 4.7%, compared to 11.8% of baseline B and 9.6% of baseline C. On IMDB-Top1000, the misalignment rate of the method of this invention is 4.2%, while that of baseline A is 16.3%, baseline B is 11.1%, and baseline C is 8.9%. As shown in Figure 6, the downstream ranking misalignment rate of the method of this invention is significantly lower than that of the three baseline methods on both datasets. In terms of construction time, the method of this invention only increases by 2.3% compared to baseline A (from 38 minutes to 38.87 minutes), while baselines B and C, due to the need to train or infer deep models, consume 4.1 times and 2.87 times more time than baseline A, respectively. In terms of interpretability score, the method of this invention scored 4.7 points (the four preset thresholds and the three corresponding fusion weights are all readable scalars that can be audited by operation and maintenance personnel one by one), baseline B scored 2.1 points (attention weights cannot be directly observed), and baseline C scored 2.8 points (the physical meaning of cross-modal similarity needs to be interpreted in conjunction with the visual language model training data).

[0058] Following Route 1, and using the characters "Lin X" and "Lao Zhang" from Example 2, statistics from Baseline A show that Lin X was sampled in all 8 source categories, appearing a total of 632 times across sources; Lao Zhang was sampled in 2 source categories, appearing a total of 2 times across sources. Baseline A assigns a confidence score of 0.81 to Lin X and 0.42 to Lao Zhang based on the logarithmic normalization of cross-source appearances. The limitation of Baseline A is that it relies solely on two spatial dimensions: whether Lao Zhang appears in a source and the total number of cross-source appearances. It fails to model the sparse distribution characteristics of Lao Zhang's single appearance in the subtitle source, leading downstream applications to easily rank Lao Zhang, who only makes a cameo appearance at the end of the film, before other medium-level characters when sorting by confidence score.

[0059] Following Route 2, and using the sequence of occurrences of "Zhou X" in the subtitle source from Example 3, the sample variance of the cross-segment unit interval sequence is 0.0034. Under this variant, the source sample variance needs to be normalized and mapped before the source stability component can be obtained; however, the total number of cross-segment units varies greatly between different sources (9842 for subtitle source and 87 for background music metadata source), making the normalized variance range incomparable between different sources. The variance, as a second-order moment, still retains a dependence on the total number of cross-segment units after normalization, leading to a systematic overestimation of the stability component for sources with small total numbers (background music metadata source, cast list source) and a systematic underestimation of the stability component for sources with large total numbers (subtitle source, still image source), making the S5 judgment threshold inconsistent between different sources. This invention uses a third-order normalized moment with sample skewness, thus eliminating the dependence on the total number scale, and cross-source comparability naturally holds.

[0060] To further verify the tolerance of this invention to missing source structures, source ablation tests were conducted on the experimental set. The first ablation involved a source lacking background music metadata (simulating a low-cost source without background music metadata), increasing the downstream sorting misalignment rate from 4.7% to 5.3%, a deterioration of 0.6 percentage points. The second ablation involved a source simultaneously lacking background music metadata and external rating links (simulating a new, unreleased source), increasing the misalignment rate to 5.9%, a deterioration of 1.2 percentage points. The third ablation involved a source simultaneously lacking posters, stills, background music metadata, and external rating links (retaining only four basic sources: subtitles, dialogue, cast list, and synopsis), increasing the misalignment rate to 7.1%, a deterioration of 2.4 percentage points, but still significantly lower than baseline A's 18.4% with all eight source types. This result demonstrates that this invention maintains its core effectiveness, namely edge discrimination capability, even with missing source structures.

[0061] The following section explains boundary conditions and anomaly handling.

[0062] Scenario 1: A candidate triple appears only once in a single source class. In this case, the triple's value in the multi-source consistency vector is set to the default value of 0.05 for the other seven positions, except for the position where the triple is set to 0.20; the decision is made according to S5. , It is classified as an edge triple with a confidence level of 0.15.

[0063] Scenario 2: A candidate triple corresponds to an entity that has multiple equivalent representations (full name, alias, nickname, etc.) in a certain source, but the entity equivalence matching rule fails to identify one of them as equivalent. In this case, the sequence of cross-segment units of the triple in the source is split into two independent sequences, each calculating its own stability component, which easily leads to an underestimation of the stability component. The mitigation method of this invention is to introduce a global dictionary preprocessing for common aliases in the S00 preprocessing step, which can cover most cases; for the very few cases not covered by the dictionary, downstream applications can re-execute S2 to S5 corrections after supplementing the alias mapping. The entire process does not require retraining any model, only re-execution of constant-level calculations.

[0064] Scenario 3: The relation type corresponding to a candidate triple is a temporal attribute (e.g., the protagonist first appears in episode 3), and its cross-segment unit sequence in the source exhibits a deliberately non-uniform distribution (all concentrated in the first 1 / 3 of the time axis). In this case, if the sample skewness formula is mechanically applied, the obtained stability component will be too low, and the temporal relation will easily be misjudged as a marginal triple. This invention can add a layer of relation type discrimination before S5: for relations with prior temporal tendency such as type identification: first appearance, last appearance, stage transition, etc., the sample skewness calculation is skipped, and the corresponding fusion weight is directly assigned according to the event confirmation criterion (evidence is extracted in 2 or more types of sources) (typically 0.65, between normal and marginal); this discrimination can be flexibly expanded by the operation and maintenance personnel through external rule files.

[0065] Scenario 4: There is a systematic timeline offset between the subtitle source and dialogue source of a film or television work (e.g., subtitle misalignment between dubbed and original versions). In this case, the segment unit number of the same line of dialogue is inconsistent in the subtitle source and the dialogue source, which can easily lead to significant differences in the stability component calculation results of the same triplet between the two types of sources. This invention does not intervene in this situation, but relies on the robustness of the multi-source consistency vector itself: the stability component is independently concatenated in S4 according to its positional encoding. Even if there is a certain degree of misalignment between the subtitle source and the dialogue source, as long as the other 6 types of sources can stably support the determination of the core triplet, S5 can still be correctly classified; when the subtitle source and the dialogue source are both severely misaligned and the support of the other sources is insufficient, the triplet degenerates into a regular or marginal triplet, which conforms to the robustness principle of conservative determination when there is insufficient evidence.

[0066] All threshold values, parameter values, number of sources, cross-segment unit definitions, and rule sets listed in the above specific embodiments are illustrative examples and are not intended to limit the scope of protection of this invention. Those skilled in the art can make adaptive adjustments to the above thresholds, parameters, source types, cross-segment unit definitions, etc., according to specific application scenarios and data characteristics without departing from the spirit and substance of this invention, and all such adjustments should be considered to fall within the scope of protection of this invention.

Claims

1. A method for automatically constructing a multimodal knowledge graph from heterogeneous film and television data sources, characterized in that, Includes the following steps: S0, access eight types of heterogeneous film and television data sources, namely, subtitle source, dialogue source, poster source, still source, background music metadata source, cast list source, plot synopsis source, and external rating link source; S1, for each type of heterogeneous video data source, divide it according to the preset cross-segment unit, and extract candidate triples from each cross-segment unit; S2, for each candidate triplet, according to the preset entity equivalence matching rule, merge the equivalent occurrences of different cross-segment units in the same type of heterogeneous film and television data source to obtain the cross-segment unit occurrence sequence of the candidate triplet in the source of the same type. S3, For each candidate triplet in the sequence of occurrence of cross-segment units in each source class, calculate the stability component. The stability component is obtained by taking the absolute value of the skewness value of the normalized cross-segment unit interval distribution and adding a preset constant inversely. S4. For each candidate triplet, its stability components on the eight sources are concatenated into a multi-source consistency vector in a preset order. S5. Based on the multi-source consistency vector of each candidate triplet, each candidate triplet is divided into one of the following: core triplet, regular triplet, or edge triplet. The specific steps are as follows: First, count the number of components in the multi-source consistency vector of each candidate triplet whose component value is not less than a preset first threshold. If the number of components is not less than the preset first threshold, then the candidate triplet is classified as a core triplet. Second, count the number of components in the multi-source consistency vector of each candidate triplet whose component value is less than the preset first threshold and not less than a preset second threshold. If the number of components is not less than the preset second threshold, then the candidate triplet is classified as a regular triplet. Other candidate triplets besides the core triplet and regular triplet are classified as edge triplets. Candidate triples are written into the multimodal knowledge graph according to their corresponding fusion weights. The corresponding fusion weight of core triples is greater than that of regular triples, and the corresponding fusion weight of regular triples is greater than that of edge triples.

2. The method according to claim 1, characterized in that, In step S1, the preset cross-segment units corresponding to the eight types of heterogeneous film and television data sources are as follows: the cross-segment unit of the subtitle source is the subtitle segment obtained by dividing the film source into scenes; the cross-segment unit of the dialogue source is the dialogue segment obtained by dividing the dialogue characters' speech; the cross-segment unit of the poster source is each poster image; the cross-segment unit of the still image source is each still image; the cross-segment unit of the background music metadata source is each background music track; the cross-segment unit of the cast list source is the cast list entry for each actor; the cross-segment unit of the synopsis source is the sentence segment obtained by dividing the synopsis into paragraphs; and the cross-segment unit of the external rating link source is each external rating page entry.

3. The method according to claim 1, characterized in that, In step S2, the preset entity equivalence matching rules are judged in the above order, including three cases: Case 1, for personal name entities, name normalization matching is adopted, that is, after removing spaces, unifying traditional and simplified characters, and unifying alias references, string equality comparison is performed. Scenario 2: For faces that contain entities, matching is performed based on the cosine similarity of the face embedding vectors being no less than a preset face similarity threshold. In case three, and in other cases, scenario-based common reference matching is used, that is, relational entities linked to by existing equivalent entities in the same cross-segment unit are classified as equivalent to that entity.

4. The method according to claim 1, characterized in that, In step S3, the stability component is calculated in sequence according to sub-steps S3a, S3b, and S3c: S3a, for the sequence of cross-segment units appearing in the source for the candidate triplet, the cross-segment unit interval sequence is obtained by the difference between the cross-segment unit numbers of two adjacent occurrences. S3b, the cross-segment unit interval sequence is normalized according to the total number of cross-segment units corresponding to the source to obtain a normalized interval sequence; S3c, calculate the sample skewness of the normalized interval sequence, take the absolute value and invert it, and then add a preset constant to obtain the stability component of the candidate triplet on the source. When the stability component exceeds the interval between zero and one, it is mapped back to the closed interval by overbound truncation.

5. The method according to claim 1, characterized in that, In step S5: the value of the preset first threshold is 0.6 to 0.9, the value of the preset second threshold is 0.3 to 0.6, and the value of the first threshold is higher than the value of the second threshold; The preset first quantity threshold is 5 to 8, the preset second quantity threshold is 3 to 6, and the value of the first quantity threshold is higher than the value of the second quantity threshold. The corresponding fusion weight of the core triplet is between 0.6 and 1.0, the corresponding fusion weight of the regular triplet is between 0.3 and 0.6, and the corresponding fusion weight of the edge triplet is between 0.05 and 0.

3.

6. The method according to claim 1, characterized in that, In step S4, the eight stability components are concatenated into a multi-source consistency vector in the order of subtitle source, dialogue source, poster source, still source, background music metadata source, cast list source, plot synopsis source, and external rating link source. For candidate triples that do not appear in a certain type of source, the stability component at the corresponding position takes a preset default value, and the preset default value is lower than the preset second threshold value.

7. The method according to claim 1, characterized in that, In step S5, writing the corresponding fusion weight into the multimodal knowledge graph specifically involves: for each candidate triple, creating a corresponding entity node, relation edge, or attribute key-value pair in the multimodal knowledge graph, and writing its corresponding fusion weight as the confidence attribute of the entity node, relation edge, or attribute key-value pair. When the same entity node or relation edge appears repeatedly in different candidate triples, the first candidate triple is selected as the final confidence level of the entity node or relation edge after sorting the corresponding fusion weight values ​​of the repeated candidate triples in descending order.

8. The method according to claim 1, characterized in that, Before step S0, a preprocessing step S00 is also included: for each type of heterogeneous video data source, cleaning is performed according to the format specification corresponding to that type of source. The cleaning includes removing duplicate entries, unifying character encoding, and decoding the binary information in the image and audio metadata into structured fields.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 8.

10. A system for automatically constructing a multimodal knowledge graph from heterogeneous film and television data sources, characterized in that, It includes six interconnected modules: a data source access module, used to access eight types of heterogeneous film and television data sources, namely, subtitle source, dialogue source, poster source, still image source, background music metadata source, cast list source, plot synopsis source, and external rating link source; The candidate triplet extraction module is used to divide each type of heterogeneous video data source into preset cross-segment units and extract candidate triplets from each cross-segment unit. The cross-segment equivalence merging module is used to merge the equivalent occurrences of different cross-segment units in the same class source for each candidate triple according to the preset entity equivalence matching rules, so as to obtain the cross-segment unit occurrence sequence of the candidate triple in the class source. The stability component calculation module is used to calculate the stability component for each candidate triplet in the cross-segment unit occurrence sequence of each source class. The stability component is obtained by taking the absolute value of the skewness value of the normalized cross-segment unit interval distribution and adding a preset constant inversely. The multi-source consistency vector concatenation module is used to concatenate the stability components of each candidate triplet on eight sources into a multi-source consistency vector in a preset order. The classification and graph writing module is used to classify each candidate triplet into one of three categories: core triplet, regular triplet, or marginal triplet, based on the multi-source consistency vector of each candidate triplet. The specific steps are as follows: First, count the number of components in the multi-source consistency vector of each candidate triplet whose component value is not less than a preset first threshold. If the number of components is not less than the preset first threshold, the candidate triplet is classified as a core triplet. Second, count the number of components in the multi-source consistency vector of each candidate triplet whose component value is less than the preset first threshold and not less than a preset second threshold. If the number of components is not less than the preset second threshold, the candidate triplet is classified as a regular triplet. Other candidate triplets besides the core triplet and regular triplet are classified as marginal triplets. Candidate triples are written into the multimodal knowledge graph according to their corresponding fusion weights. The corresponding fusion weight of the core triple is greater than that of the regular triple, and the corresponding fusion weight of the regular triple is greater than that of the edge triple.

Images