Policy knowledge graph assisted multi-source domain corpus fusion method

Abstract

The invention relates to the technical field of natural language information fusion, and discloses a policy knowledge graph assisted multi-source domain corpus fusion method. The method comprises the following steps: performing deep structure analysis on multi-field original corpora to generate a structured corpus set; and performing structural evaluation on the corpus by using a pre-constructed policy knowledge graph, identifying a mapping relationship and difference with the graph, and generating an analysis result with difference measurement. Policy impact factors are extracted by analyzing a real-time policy text stream and are dynamically embedded into the knowledge graph, and a policy enhanced knowledge graph structure is generated. And under the guidance of the structure, heterogeneous normalization and collaborative alignment based on a graph semantic path are performed on multi-source corpora, cross-domain consistency check and conflict resolution are performed, and finally a consistency fusion corpus is generated. According to the method, the fusion process is structurally guided and updated in real time through dynamic policy knowledge, and the structural consistency and policy timeliness of cross-domain corpus fusion are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language information fusion, in particular to a policy knowledge graph assisted multi-source domain corpus fusion method. BACKGROUND

[0002] In the field of multi-source heterogeneous information processing, integrating text corpora from different independent domains is a basic and key task. Existing technologies usually rely on statistical-based text similarity calculation, pre-trained language model semantic vector matching, or ontology alignment methods based on artificial rules to complete the association and fusion of corpora. These methods focus on finding commonalities from the lexical, syntactic or shallow semantic features of the corpus itself, and the fusion process lacks macro guidance from the high-level knowledge structure of a specific domain. When dealing with multi-domain corpora closely related to policies, this content-driven fusion method is difficult to ensure that the fusion results conform to the inherent logical association and concept system under the policy background, resulting in limitations in semantic consistency and domain adaptability of the fused corpus.

[0003] In the research of introducing knowledge graphs to provide structured knowledge guidance, the common approach is to link or annotate corpora with a statically constructed knowledge graph. However, the policy environment has significant dynamics and timeliness, and new policy texts are constantly being released, with new orientations, new focuses and new concepts evolving continuously. A static knowledge graph cannot absorb these dynamic changes in policy information in a timely manner, and its nodes and relationships will gradually become disconnected from the reality of the policy context. Therefore, the evaluation and alignment of corpora based on outdated graphs cannot accurately reflect the semantic influence and relationship constraints of the latest policy context on multi-source domain concepts, making the fusion work lag behind the needs of actual policy analysis. SUMMARY

[0004] The purpose of the present application is to provide a policy knowledge graph assisted multi-source domain corpus fusion method to solve the problems raised in the background art.

[0005] To achieve the above purpose, the present application provides a policy knowledge graph assisted multi-source domain corpus fusion method, which comprises: Obtaining a set of original corpora from multiple independent domains, and performing deep structure feature analysis on the set of original corpora of each domain to generate a domain structured corpus set containing semantic units, entity labels and association strength; Receiving the domain structured corpus set and performing structural evaluation based on a pre-constructed policy knowledge graph, identifying the mapping relationship and differences between corpus elements and graph nodes and edges, and generating a graph alignment analysis result with mapping labels and difference measures; extracting the difference measure in the graph alignment analysis result, and introducing a real-time policy text stream to analyze policy factors, embedding the analyzed policy influence factors into the corresponding nodes and edges of the knowledge graph to generate a policy-enhanced knowledge graph structure; According to the policy-enhanced knowledge graph structure, performing heterogeneous normalization processing on the multi-source domain structured corpus set to generate a normalized corpus vector set; Performing multi-source collaborative alignment operation on the normalized corpus vector set, calculating the semantic similarity and correlation between vectors based on the semantic path of the knowledge graph, and generating a fusion corpus cluster; Performing cross-domain consistency verification and conflict resolution on the fusion corpus cluster, modifying or merging the marked content according to the predefined resolution rules to generate a consistent fusion corpus library.

[0006] Preferably, the specific steps of the deep structure feature analysis include: Performing word segmentation, part-of-speech tagging, and named entity recognition on the original corpus set to extract basic semantic units and entity objects; Performing dependency syntax analysis and semantic role labeling on the identified semantic units and entity objects to construct a syntax relationship and logical relationship network within the sentence; Based on the pre-trained domain word vector model, calculate the vector representation of each element in the semantic unit, entity object and relationship network and the correlation strength score between them; Integrate the semantic units, entity objects, relationship networks, vector representations and correlation strength scores, and encapsulate them by domain classification to form the domain structured corpus set.

[0007] Preferably, the domain structured corpus set is received, and based on the pre-constructed policy knowledge graph, its structure is evaluated, the mapping relationship and difference between corpus elements and graph nodes and edges are identified, and a graph alignment analysis result with mapping marks and difference measures is generated, including: Comparing the entity objects and semantic relationships in the domain structured corpus set with the nodes and edges in the policy knowledge graph one by one; When the entity object or semantic relationship can be directly matched to the graph node or edge, record as complete mapping, and mark the mapping path; When the entity object or semantic relationship has semantic similarity but not complete consistency with the graph node or edge, record as fuzzy mapping, and calculate the cosine distance in the vector space as the difference measure; When the entity object or semantic relationship is completely missing in the graph, record as unmapped, and mark as an element to be expanded; Summarize all comparison results to form the graph alignment analysis result containing the complete mapping set, the fuzzy mapping set and its difference measure, and the unmapped set.

[0008] Preferably, the introduction of real-time policy text stream for policy factor analysis includes: Continuous crawling and monitoring of real-time policy text streams from designated official sources; Extracting and structuring key policy provisions, subjects, objects and constraints from the real-time policy text stream; Matching the structured policy provisions with the concept nodes in the policy knowledge graph to identify the graph areas directly affected by the policy; Convert the impact strength, direction and timeliness of the policy into computable weight factors and embed them as attributes into the corresponding graph nodes and edges; Output the policy-enhanced knowledge graph structure embedded with policy weight factors.

[0009] Preferably, the specific steps of the heterogeneous normalization process include: Define a unified metadata schema and semantic type system for the domain structured corpus from different sources; Convert the original entity labels and relationship types in the corpus to the unified semantic type system according to the mapping dictionary; Map the non-standardized vector representation of corpus elements to a unified standardized semantic vector space through a shared projection matrix; Add unified metadata identifiers of source, processing timestamp and version information to all corpus elements to generate the normalized corpus vector set in standard format.

[0010] Preferably, the specific steps of the multi-source collaborative alignment operation include: Use the policy-enhanced knowledge graph structure as a semantic bridge to calculate the shortest semantic path distance between any two corpus vectors in the normalized corpus vector set; According to the shortest semantic path distance, construct a semantic similarity matrix and a semantic correlation matrix between corpus vectors; Use a density-based clustering algorithm to combine the semantic similarity matrix and the semantic correlation matrix to preliminarily group the corpus vectors; Within each preliminary grouping, re-order and re-organize the corpus vectors according to semantic correlation to form the fusion corpus cluster with clear themes.

[0011] Preferably, the specific steps of the cross-domain consistency check and conflict resolution include: Check whether there are factual contradictions or numerical conflicts in the descriptions of the same entity or event in different source corpora within the fusion corpus cluster; Check whether there is unnecessary semantic information overlap between different corpus segments; For the identified factual contradictions, priority rules are set according to the authority, timeliness and fitting degree of the corpus source, high-priority corpora are retained, and low-priority corpora are marked or modified; For the identified semantic overlaps, a fusion algorithm based on sentence boundaries is used for deduplication and smooth merging; The results after verification and resolution are integrated into a logically consistent consistency fusion corpus.

[0012] Preferably, the method further comprises: Based on the consistency fusion corpus, the policy-enhanced knowledge graph structure is iteratively updated, the fused new semantic relationships and entities are injected into the graph, the connections between nodes are expanded, and the weights of edges are optimized, to generate an updated dynamic knowledge graph; Using the updated dynamic knowledge graph, the consistency fusion corpus is quality evaluated and dynamically screened, the fitting degree of the corpus unit and the graph structure is calculated, low-quality or outlier corpus units are filtered according to the fitting degree threshold, and a refined fusion corpus result is generated; A traceability and feedback link is established between the refined fusion corpus result and the original multi-field corpus, the transformation history of each corpus unit in the fusion process is recorded, and the fusion rules are dynamically calibrated according to the transformation history, to generate a calibrated fusion rule set; Based on the calibrated fusion rule set and the updated dynamic knowledge graph.

[0013] Preferably, the iterative updating of the policy-enhanced knowledge graph structure based on the consistency fusion corpus comprises: New entities, attributes and inter-entity relationships are extracted from the consistency fusion corpus; The new entity is taken as a candidate node, and its semantic association with existing nodes in the graph is calculated. If the association degree exceeds the threshold, a connection is established, otherwise a new node is created; The new inter-entity relationship is taken as a candidate edge, and its weight is initialized using the co-occurrence frequency and context strength in the corpus; Using global statistical information from the consistency fusion corpus, the weights of existing edges in the graph are recalibrated; The updated dynamic knowledge graph containing new nodes, new edges and updated weights is generated.

[0014] Preferably, the specific steps of quality evaluation and dynamic screening comprise: The overall semantic fitting degree of each corpus unit in the consistency fusion corpus and the updated dynamic knowledge graph is calculated, which is determined by the matching degree of entities within the unit and the nodes in the graph, and the matching degree of relationships and edges in the graph; Set a dynamic quality threshold, which is adjusted according to the target of the fusion task and the overall distribution of the corpus; Screen out the corpus units with high semantic fit degree higher than the quality threshold to form a core corpus set; The corpus units with a semantic fit degree lower than the quality threshold are analyzed and classified as correctable noise corpus or irrelevant outlier corpus, and the correctable noise corpus is corrected under the guidance of the atlas; The core corpus set and the corrected corpus are merged, and the output is the refined fusion corpus result.

[0015] Compared with the prior art, the beneficial effects of the present application are: By pre-building a knowledge graph covering policy concepts, entities and their relationships, and using it as a benchmark structure to identify mapping relationships and differences of multi-source domain structured corpus, an analysis result with quantitative difference measurement is generated. This technology changes the driving mechanism of corpus fusion from the traditional two-by-two similarity calculation between corpora to the evaluation and diagnosis based on the external authoritative policy knowledge structure as the ruler. This makes the fusion process able to clearly identify which corpus elements conform to the policy knowledge framework and which have deviations or omissions, providing a quantifiable adjustment basis based on structural consistency for subsequent fusion, rather than relying only on semantic proximity in a statistical sense.

[0016] Real-time policy text stream is introduced, and policy-oriented, key areas, emerging terms and other policy influencing factors are extracted from it through analysis technology, and these dynamic factors are embedded into the original knowledge graph in the form of node attribute enhancement, relationship weight adjustment or new association edge. This technology breaks the limitation of the knowledge graph being static in the application of corpus fusion, and constructs a dynamic graph structure that can be updated synchronously with the evolution of the policy. The enhanced graph guided fusion makes the normalization, alignment and conflict resolution process of multi-source domain corpus respond to the latest policy context in real time, ensuring that the internal logic of the corpus formed by the fusion is consistent with the current policy focus and expression method, and improving the timeliness and policy relevance of the fusion result. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The working principle diagram of the policy knowledge graph assisted multi-source domain corpus fusion method described in the present application; Figure 2 The flowchart for deep structure feature analysis; Figure 3 The flowchart for policy factor analysis; Figure 4 The grouped column chart for quality comparison before and after multi-source corpus fusion; Figure 5 The grouped column chart for quality comparison before and after refining the corpus. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application.

[0019] Please refer to Figure 1 The present application provides a policy knowledge graph assisted multi-source domain corpus fusion method, which comprises: obtaining original corpus sets from multiple independent domains, such as finance, technology, industry, etc., and performing deep structure feature analysis on the original corpus set of each domain. The analysis process extracts semantic units, entity labels in the corpus, and calculates the correlation strength between them, and finally generates a structured domain structured corpus set. Receive the domain structured corpus set, and based on a pre-constructed policy knowledge graph containing policy related concepts and relationships, perform structural evaluation on it. This evaluation identifies the mapping relationship and difference between the elements in the corpus and the nodes and edges in the graph, and outputs the graph alignment analysis result with mapping marks and difference metrics.

[0020] Extract the difference metrics in the graph alignment analysis result, and introduce real-time policy text stream for policy factor analysis. The analysis process extracts policy impact factors from the policy text, and embeds these factors in the form of weights into the corresponding nodes and edges of the knowledge graph, thereby generating a policy enhanced knowledge graph structure reflecting the latest policy direction. According to the policy enhanced knowledge graph structure, perform heterogeneous normalization processing on the structured corpus sets from different domains. This processing maps corpora of different formats and semantic systems to a unified vector space and type system, generating a normalized corpus vector set. Perform multi-source collaborative alignment operation on the normalized corpus vector set. This operation takes the policy enhanced knowledge graph as a semantic bridge, calculates the semantic similarity and relevance between vectors, and performs clustering and reorganization based on this, generating a fusion corpus cluster with internal theme consistency. Perform cross-domain consistency test and conflict resolution on the fusion corpus cluster. Check and process possible factual contradictions and semantic overlaps between different source corpora, modify or merge the marked content according to the predefined resolution rules, and finally generate a consistent fusion corpus library with logical consistency and information complementarity.

[0021] Embodiment 1: Please refer to Figure 2The original corpus set is segmented, word tagged, and named entity recognized to extract basic semantic units and entity objects. The identified semantic units and entity objects are subjected to dependency syntax analysis and semantic role labeling to construct the internal syntax and logic relationship network of the sentence. Based on the pre-trained domain word vector model, the vector representation of each element in the semantic unit, entity object, and relationship network and the correlation strength score between them are calculated. The semantic units, entity objects, relationship networks, vector representations, and correlation strength scores are integrated, packaged according to the domain classification, and formed into the domain structured corpus set. The domain structured corpus set is received, and based on the pre-constructed policy knowledge graph, it is subjected to structural evaluation, the mapping relationship and difference between the corpus elements and the graph nodes and edges are identified, and the graph alignment analysis result with mapping marks and difference metrics is generated. The process includes comparing the entity objects and semantic relationships in the domain structured corpus set with the nodes and edges in the policy knowledge graph one by one. When the entity objects or semantic relationships can be directly matched to the graph nodes or edges, they are recorded as complete mapping and the mapping path is marked. When the entity objects or semantic relationships are semantically similar but not completely consistent with the graph nodes or edges, they are recorded as fuzzy mapping and the cosine distance in the vector space is calculated as the difference metric. When the entity objects or semantic relationships are completely missing in the graph, they are recorded as unmapped and marked as elements to be expanded. All comparison results are summarized to form the graph alignment analysis result containing the complete mapping set, the fuzzy mapping set and its difference metric, and the unmapped set.

[0022] In a specific implementation, the original corpus set comes from multiple independent domains such as finance, technology, and industry. The deep structure feature analysis first segments, word tags, and named entity recognizes the original corpus set in the finance domain to extract basic semantic units and entity objects. Named entity recognition can identify entity objects such as “listed company”, “carbon neutral”, and “R&D investment”. Then the identified semantic units and entity objects are subjected to dependency syntax analysis and semantic role labeling to construct the internal syntax and logic relationship network of the sentence, and the correlation strength score between them is calculated. The correlation strength score is calculated by the co-occurrence and dependency relationship of the semantic unit vector in a specific context window. The semantic units, entity objects, relationship networks, vector representations, and correlation strength scores obtained by all analyses are integrated, packaged according to the finance domain classification, and formed into the domain structured corpus set in the finance domain. The original corpus set in the technology, industry, and other domains is repeatedly subjected to the same deep structure feature analysis process to finally generate the domain structured corpus set in all corresponding domains.

[0023] In a specific implementation, after receiving the generated multiple domain structured corpus, a structural evaluation is performed based on the pre-constructed policy knowledge graph. The semantic relationship "Enterprise A leads the research and development of technology B" in the structured corpus of the science and technology field is compared with the edges in the policy knowledge graph. The relationship "leading research and development" is similar to but not completely consistent with the "leading innovation" edge in the graph, and is recorded as a fuzzy mapping. The cosine distance of "leading research and development" and "leading innovation" in the vector space is calculated as a difference measure, for example, the calculated cosine distance value is 0.15. The new entity "zero-carbon factory" identified in the structured corpus of the industrial field is compared with the policy knowledge graph. If no direct corresponding or highly similar node is found in the graph, "zero-carbon factory" is recorded as unmapped and marked as an element to be expanded. The comparison results of all domains are summarized to form a graph alignment analysis result including the complete mapping set, the fuzzy mapping set and its difference measure, and the unmapped set.

[0024] One embodiment of calculating the association strength score between a semantic unit and an entity object can be performed by the following formula: Wherein, the symbol represents the association strength score between a semantic unit and an entity object , the symbols and represent the vector representation of the semantic unit and the entity object , respectively, the function calculates the cosine similarity of the two vectors, the symbol represents the number of times the semantic unit and the entity object co-occur in the corpus, the symbols and represent the total number of independent occurrences of the semantic unit and the entity object , respectively, and the symbol is a weight coefficient between 0 and 1, used to adjust the contribution proportion of vector similarity and co-occurrence statistical information.

[0025] In some embodiments, the difference measure when computing the fuzzy mapping can adopt Euclidean distance or Manhattan distance in vector space. The calculation of the difference measure needs to be set in coordination with the judgment threshold of complete mapping, for example, setting the cosine similarity above 0.9 as complete mapping, and the similarity between 0.7 and 0.9 as fuzzy mapping. Optionally, when constructing the syntactic and logical relationship network within the sentence, in addition to dependency syntax analysis and semantic role labeling, discourse structure analysis can be introduced to capture the semantic association across sentences. Optionally, the pre-trained domain word vector model can be trained using a model based on the Transformer architecture. It can be understood that the complete mapping set in the atlas alignment analysis result can be directly used for subsequent fusion, the fuzzy mapping set needs to be further processed in combination with the difference measure, and the unmapped set prompts that the policy knowledge graph needs to be potentially expanded. It can be understood that the quality of deep structure feature analysis directly determines the accuracy of the domain structured corpus set, and then affects the accuracy of subsequent structural evaluation.

[0026] Embodiment 2: Referring to Figure 3 , continuously crawling and monitoring real-time policy text streams from designated official sources, extracting and structuring key policy provisions, subjects, objects, and constraint conditions from the real-time policy text streams, associating and matching the structured policy provisions with the concept nodes in the policy knowledge graph, identifying the graph area directly affected by the policy, converting the influence strength, action direction, and timeliness of the policy into computable weight factors, and embedding them as attributes into the corresponding graph nodes and edges, outputting the policy-enhanced knowledge graph structure embedded with the policy weight factors. The specific steps of heterogeneous normalization processing include defining a unified metadata schema and semantic type system for the domain structured corpus sets from different sources, converting the original entity labels and relationship types in the corpus to the unified semantic type system according to the mapping dictionary, mapping the non-standardized vector representation of the corpus elements to the unified standardized semantic vector space through a shared projection matrix, adding unified metadata identifiers of source, processing timestamp, and version information to all corpus elements, and generating the normalized corpus vector set in standard format.

[0027] In practical implementation, real-time policy text streams are introduced for policy factor analysis, continuously crawling and monitoring real-time policy text streams from designated official sources. The impact intensity, direction of action, and timeliness of policies are transformed into calculable weight factors. For example, a policy impact weight factor is set for the "core industries of the digital economy" node. The weight value is calculated based on the difference between the policy document level, specific quantitative indicators, and the current time, and the weight factor is embedded as an attribute into the corresponding graph nodes and edges. The output is a policy-enhanced knowledge graph structure with embedded policy weight factors. In this structure, the attribute set of the "core industries of the digital economy" related nodes and connecting edges is supplemented with information such as "policy weight: 0.85, effective date: 2025". After completing policy factor analysis and generating a policy-enhanced knowledge graph structure, heterogeneous normalization processing is performed on multi-source domain structured corpora based on this structure. A unified metadata model and semantic type system are defined for the different source domain structured corpora. The unified metadata model includes fields such as "corpus ID," "original domain," "processing timestamp," and "version number," while the unified semantic type system includes types such as "macro concept," "micro entity," "positive relation," and "negative relation." The original entity labels and relation types in the corpora are converted to the unified semantic type system using a pre-generated mapping dictionary. For example, the original entity label "credit risk" in the financial domain structured corpus and "technological risk" in the technology domain structured corpus are both converted to the unified type "operational risk" using the mapping dictionary.

[0028] In practice, the core step of heterogeneous normalization is to map the non-standardized vector representations of corpus elements to a unified space. This mapping is achieved through a shared projection matrix, which maps the non-standardized vector representations of corpus elements to a unified, standardized semantic vector space. Vectors from the financial domain might be trained on a financial terminology corpus in their original vector space, while vectors from the technology domain might be trained on a corpus of scientific papers. The shared projection matrix learns a transformation that aligns different source vector spaces to a common space anchored by policy-enhanced knowledge graph node vectors. Unified metadata identifiers, including source, processing timestamp, and version information, are added to all corpus elements.

[0029] The process of mapping non-normalized vectors to a normalized semantic vector space can be characterized by the following formula: Where: symbol Indicates from the Standardized semantic vectors obtained by mapping corpus elements from various domains, symbols Represents the shared projection matrix, symbol This indicates that the corpus element is in its original... non-standardized vector representation in a domain vector space, denoted as denotes a bias vector. A shared projection matrix and bias vector are learned by optimizing an objective function that requires the mapped normalized vectors of semantically similar corpus elements from different domains to be as close as possible in the space.

[0030] In some embodiments, the construction of the mapping dictionary can be semi-automatically generated based on the node type system of the policy-enhanced knowledge graph, supplemented by manual review and correction. The shared projection matrix needs to be trained and learned using a part of the cross-domain aligned corpus. Optionally, the unified metadata schema can also include extended fields such as corpus original confidence and policy correlation degree. Optionally, the dimension of the normalized semantic vector space can be consistent with the dimension of the node vector in the policy-enhanced knowledge graph. It can be understood that the policy factor analysis enables the knowledge graph to have a dynamic policy weight attribute, providing a policy sensitivity measurement benchmark for subsequent fusion. Heterogeneous normalization is a prerequisite for subsequent cross-domain corpus vector comparability calculation and collaborative alignment, and the accuracy of its mapping directly affects the fusion effect.

[0031] In some embodiments, the construction of the mapping dictionary can be semi-automatically generated based on the node type system of the policy-enhanced knowledge graph, supplemented by manual review and correction. The shared projection matrix needs to be trained and learned using a part of the cross-domain aligned corpus. Optionally, the unified metadata schema can also include extended fields such as corpus original confidence and policy correlation degree. Optionally, the dimension of the normalized semantic vector space can be consistent with the dimension of the node vector in the policy-enhanced knowledge graph. It can be understood that the policy factor analysis enables the knowledge graph to have a dynamic policy weight attribute, providing a policy sensitivity measurement benchmark for subsequent fusion. Heterogeneous normalization is a prerequisite for subsequent cross-domain corpus vector comparability calculation and collaborative alignment, and the accuracy of its mapping directly affects the fusion effect.

[0032] ​In a specific implementation, a multi-source collaborative alignment operation is performed on the normalized corpus vector set according to the policy-enhanced knowledge graph structure, taking the policy-enhanced knowledge graph structure as a semantic bridge, calculating the shortest semantic path distance between any two corpus vectors in the normalized corpus vector set. For example, there are multiple nodes and connecting edges such as "artificial intelligence", "algorithm", "data security", "policy supervision" in the policy-enhanced knowledge graph structure, and the shortest semantic path distance between the corpus vector describing "artificial intelligence ethics" and the corpus vector describing "data privacy protection" is calculated. The path may pass through nodes and edges such as "artificial intelligence" -> "dependence" -> "data" -> "association" -> "privacy protection", and the sum of the reciprocals of the weights of all edges on the path is taken as the measurement value of the shortest semantic path distance. According to the shortest semantic path distance calculated for all corpus vector pairs, an N x N matrix is constructed as a semantic similarity matrix, and the matrix element value is obtained by transforming the shortest semantic path distance through a negative exponential function, with a value range of 0 to 1. At the same time, a semantic correlation matrix is constructed, and the element value of the semantic correlation matrix is not only considered in terms of the shortest path distance, but also combined with the number of common neighbor nodes of the two corpus vectors in the policy-enhanced knowledge graph for calculation. Using a density-based clustering algorithm, the semantic similarity matrix and the semantic correlation matrix are used as input distance measurement criteria, and all corpus vectors are preliminarily grouped. The density threshold parameter is set according to the data distribution, and preliminary grouping will produce multiple core corpus groups and several noise points. Within each preliminary grouping, the corpus vectors are rearranged in descending order according to the semantic correlation values between the corpus vectors, and the rearranged corpus fragments are reorganized semantically, forming fused corpus clusters with clear themes through connecting word merging or abstract generation.

[0033] In a specific implementation, after generating the fused corpus cluster, cross-domain consistency verification and conflict resolution are performed on the fused corpus cluster. Within the fused corpus cluster, it is checked whether there is factual contradiction or numerical conflict in the description of the same entity or event by different source corpora. For example, the corpus from a certain financial news describes that "A company's new energy vehicle sales increased by 80% this year", while the corpus from a certain industry report describes that "A company's new energy vehicle sales increased by 65% this year", and the system identifies that there is a conflict in the numerical value of the growth of the entity "A company's new energy vehicle sales". It is checked whether there is unnecessary semantic information overlap between different corpus segments. For example, two segments of corpora from different technology media both describe the same technical principle of "battery energy density improvement" in detail, and the system identifies that there is a large amount of semantic information overlap. For the identified factual contradiction, according to the predefined resolution rules, the resolution rules set the authority, timeliness and fit degree with the knowledge graph of the corpus source as the priority judgment dimensions, and the corpus description from the authoritative statistical agency and the more timely one is retained, and the other corpus is marked or the numerical value is corrected based on the reliable source. For the identified semantic overlap, a sentence boundary-based fusion algorithm is used for deduplication and smooth merging. The algorithm compares the vector similarity of overlapping sentences, retains the sentence version with the most complete information, and uses reference resolution and coherence connection technology to smoothly merge the information from different corpus segments into a logically coherent paragraph. All corpus segments after verification and resolution are integrated and output as a logically consistent, non-redundant and consistent fused corpus library.

[0034] Computing semantic correlation between two corpus vectors p and q based on a knowledge graph One embodiment can be performed by the following formula: Wherein: symbol represents the semantic correlation score between corpus vector p and corpus vector q, symbol represents the shortest semantic path distance between the concept nodes corresponding to corpus vector p and q in the policy-enhanced knowledge graph, function is a negative exponential function, symbol and respectively represent the first-order neighborhood node set of the concept nodes corresponding to corpus vector p and q in the policy-enhanced knowledge graph, symbol is a balance coefficient for adjusting the weight of the two contributions.

[0035] See Figure 4This is a grouped column chart for quality comparison before and after multi-source corpus fusion. The core is to show the changes in the corpus on the four indicators of "semantic fit", "redundancy rate", "conflict rate" and "completeness". It belongs to data visualization chart. The score after fusion (about 0.9) is significantly higher than that before fusion (about 0.72), indicating that the matching degree of corpus and knowledge graph has been greatly improved; the score after fusion (about 0.08) is much lower than that before fusion (about 0.35), reflecting the effect of semantic deduplication; the score after fusion (about 0.05) is much lower than that before fusion (about 0.28), indicating that the conflict resolution is effective; the score after fusion (about 0.95) is slightly higher than that before fusion (about 0.85), which optimizes the quality while retaining the information integrity. It is commonly used to show the effect of data processing / fusion tasks. By comparing the changes in core indicators, the value of algorithm / process can be intuitively reflected. It is commonly used in academic reports, project achievement reports and other scenarios.

[0036] In embodiment 4, the consistent fusion corpus drives the policy-enhanced knowledge graph structure to update iteratively. New entities, attributes and entity relationships are extracted from the consistent fusion corpus. New entities are used as candidate nodes, and their semantic association with existing nodes in the graph is calculated. If the association degree exceeds the threshold, a connection is established, otherwise a new node is created. New entity relationships are used as candidate edges, and their weights are initialized using co-occurrence frequency and context strength in the corpus. Global statistical information from the consistent fusion corpus is used to recalibrate the weights of existing edges in the graph, generating an updated dynamic knowledge graph containing new nodes, new edges and updated weights. The updated dynamic knowledge graph is used to evaluate the quality of the consistent fusion corpus and dynamically filter it. The fit degree of the corpus unit and the graph structure is calculated, and low-quality or outlier corpus units are filtered based on the fit degree threshold to generate a refined fusion corpus result. A traceability and feedback link is established between the refined fusion corpus result and the original multi-domain corpus, the transformation history of each corpus unit in the fusion process is recorded, and the fusion rules are dynamically calibrated based on the transformation history to generate calibrated fusion rule sets. Based on the calibrated fusion rule sets and the updated dynamic knowledge graph, the system forms a closed-loop optimization mechanism.

[0037] In a specific implementation, based on the consistent fusion corpus driven policy enhanced knowledge graph structure is updated iteratively, the new entity, attribute and entity relationship are extracted from the consistent fusion corpus, for the new concept which cannot find the high correlation degree node in the existing graph, then create an independent new node. The new entity relationship is "applied to" as a candidate edge, connecting the "hydrogen fuel cell" node and the "inland shipping" node, and using the co-occurrence frequency and context strength of "hydrogen fuel cell" and "inland shipping" in the consistent fusion corpus to initialize the weight of the "applied to" edge. Using the global statistical information from the consistent fusion corpus, the weight of the existing edge in the graph is recalibrated, for example, the original "support" edge in the graph connects the "industrial policy" node and the "new energy vehicle" node, and the weight is 0.6, according to the consistent fusion corpus statistics, it is found that the corpus frequency of mentioning this support relationship has increased significantly and the context sentiment is more positive, then the weight of the "support" edge is recalibrated to 0.85. The updated dynamic knowledge graph containing new nodes, new edges and updated weights is generated, refer to table 1, which shows part of the changes of the knowledge graph before and after one iteration update.

[0038] Table 1: node and edge information table before and after the local update of the knowledge graph In a specific implementation, the consistency fusion corpus is quality evaluated and dynamically screened using the updated dynamic knowledge graph, the overall semantic fit degree of each corpus unit in the consistency fusion corpus and the updated dynamic knowledge graph is calculated, the fit degree is jointly determined by the matching degree of entities in the unit and the nodes of the graph, the matching degree of relationships and the edges of the graph, low-quality or outlier corpus units are filtered according to the fit degree threshold, a refined fusion corpus result is generated, the initial value of the dynamic fit degree threshold is set to 0.7, the corpus units with a fit degree higher than 0.7 are screened out to form a core corpus set, the corpus units with a fit degree lower than 0.7 are analyzed and classified as correctable noise corpus or irrelevant outlier corpus, for example, a corpus unit has a low fit degree due to unrecognized entity alias and is classified as correctable noise corpus and is corrected to "lithium-ion power battery" node in the graph under the guidance of the graph; another corpus unit discusses "classical literature" which is completely irrelevant to the current graph field and is classified as irrelevant outlier corpus and is directly removed. A traceability and feedback link is established between the refined fusion corpus result and the original multi-field corpus, the complete transformation history of each corpus unit from original parsing, normalization, alignment to screening is recorded, and the fusion rules are dynamically calibrated according to the transformation history, for example, if the number of corpus units filtered due to "entity alias" problem significantly increases recently, the generation rules of the entity mapping dictionary are dynamically calibrated, the frequency of synonym mining is increased, and the calibrated fusion rule set is generated. Based on the calibrated fusion rule set and the updated dynamic knowledge graph, the system forms a closed loop for sustainable optimization of knowledge graph and corpus quality.

[0039] Computing semantic association degree of candidate new entity with existing nodes in the graph One embodiment of the semantic association degree of the candidate new entity and the existing nodes in the graph can be calculated by the following formula: wherein the symbol represents the semantic association degree of the candidate new entity and the existing nodes in the graph , the function represents the vocabulary set of the entity or node name and its common alias, the symbols and are the vocabularies from the vocabulary set of the candidate new entity and the vocabulary set of the existing nodes , the function calculates the cosine similarity of the vocabulary and the vocabulary in the pre-trained word vector model.

[0040] In some embodiments, the consistency threshold can be inversely deduced and dynamically adjusted according to the target size of the refined fusion corpus result. It can be understood that establishing the traceability and feedback link makes the fusion process interpretable.

[0041] In some embodiments, the consistency threshold can be inversely deduced and dynamically adjusted according to the target size of the refined fusion corpus result. It can be understood that establishing the traceability and feedback link makes the fusion process interpretable.

[0042] In a specific implementation, the overall semantic consistency of each corpus unit in the consistency fusion corpus and the updated dynamic knowledge graph is calculated, and the semantic consistency is jointly determined by the matching degree of the entity in the unit and the node in the graph, and the matching degree of the relationship and the edge in the graph. For a specific corpus unit, its content describes "a company launches a solid-state battery product, and the charging efficiency is significantly improved", the system identifies that the corpus unit contains entities "solid-state battery" and "charging efficiency", and the relationship "improvement". In the updated dynamic knowledge graph, the "solid-state battery" node exists, the "charging efficiency" exists as an attribute of the "battery" node, and the relationship "improvement" can be matched with the "performance improvement" edge in the graph. The system calculates the entity matching degree and the relationship matching degree respectively, and the entity matching degree is based on the semantic similarity of the entity name and the graph node name, and the consistency of the entity context and the graph node context; the relationship matching degree is based on the semantic similarity of the relationship description and the graph edge definition. Combining the entity matching degree and the relationship matching degree, the overall semantic consistency of this corpus unit is obtained by weighted summation according to the preset weight, for example, the consistency score is calculated to be 0.82. A dynamic quality threshold is set, and the quality threshold is adjusted according to the target of the fusion task and the overall distribution of the corpus. In the initial stage, the quality threshold is set to 0.7, when the consistency score distribution of the consistency fusion corpus this time presents right skewed state, most of the corpus unit scores are higher than 0.75, and the fusion task this time needs more comprehensive corpus coverage, the system will dynamically lower the quality threshold to 0.65, otherwise if high-precision corpus is needed, the threshold can be adjusted to 0.75. The system selects the corpus unit with a semantic consistency higher than the dynamic quality threshold to form a core corpus set, for example, when the consistency threshold is 0.65, all corpus units with a consistency score greater than or equal to 0.65 are retained and included in the core corpus set.

[0043] In a specific implementation, the corpus units with a semantic fit score lower than the dynamic quality threshold are analyzed and classified as correctable noise corpus or irrelevant outlier corpus. One corpus unit mentions "sodium-ion battery energy density breakthrough", with a fit score of 0.60, which is lower than the threshold of 0.65. The system analyzes the reason for the low score and finds that the identified entity "sodium-ion battery" has no direct corresponding node in the updated dynamic knowledge graph, but has a potential association with the existing "new battery technology" node. This corpus unit is classified as correctable noise corpus. Another corpus unit discusses "traditional fuel vehicle second-hand car market trend", with a fit score of 0.20. The system analyzes and finds that its content has very weak association with the core fields such as "new energy" and "intelligent manufacturing" focused on by the knowledge graph, and is classified as irrelevant outlier corpus. The units classified as correctable noise corpus are corrected under the guidance of the graph. For the "sodium-ion battery" corpus unit, the system finds the most similar node in the updated dynamic knowledge graph and discovers that "sodium-ion battery" is highly similar to the "lithium-ion battery" sibling node concept under the "new battery technology" node, and has a strong correlation with the "energy storage material" node in the co-occurrence context in the corpus. The system suggests creating a new "sodium-ion battery" node in the knowledge graph as a child node of "new battery technology" and establishing a "usage" relationship with the "energy storage material" node. At the same time, the entity annotation in the corpus unit is corrected by linking it to the newly created "sodium-ion battery" node. After correction, the fit score of the corpus unit is improved to 0.78. The core corpus set and the corrected corpus are merged, and the output is the refined fusion corpus result. The semantic consistency of all corpus units in the result with the updated dynamic knowledge graph is significantly higher than that of the original consistent fusion corpus.

[0044] The overall semantic fit score of a corpus unit a with the updated dynamic knowledge graph G is calculated as follows: One embodiment can be calculated by the following formula: Wherein the symbol represents the overall semantic fit score of the corpus unit a, and the symbol represents the set of entities identified in the corpus unit a, and the symbol represents the set of relationships identified in the corpus unit a, and the function calculates the best matching degree score of entity e with all nodes in the knowledge graph G, and the function calculates the best matching degree score of relationship r with all edges in the knowledge graph G. The formula calculates the average value of the sum of entity matching degree and the sum of relationship matching degree as the overall semantic fit.

[0045] In some embodiments, the dynamic quality threshold can be automatically adjusted according to the target capacity proportion of the core corpus, for example, if it is set to retain the top 80% of the corpus, the threshold is selected to match the 20th percentile of the coherence score. It can be understood that the correctable noise corpus usually contains new concepts or aliases not covered by the atlas, while the irrelevant outlier corpus belongs to out-of-domain noise. Optionally, the correction process of the correctable noise corpus can be cross-verified in combination with external domain dictionaries or ontology libraries. Optionally, the determination of irrelevant outlier corpus can be assisted by determining the average semantic distance between the irrelevant outlier corpus and the corpus in the core corpus.

[0046] Referring to Figure 5 This is a grouped column chart comparing the quality of the corpus before and after refinement, which is the visualization result of the "quality evaluation and dynamic screening stage" in the policy knowledge graph corpus fusion process, and is used to show the optimization effect of the corpus refinement operation on the quality indicators. The overall semantic coherence after refinement (green column, about 0.85) is significantly higher than that before refinement (orange column, about 0.72), indicating that the matching degree of the corpus and the knowledge graph has been greatly improved; the proportion of core corpus after refinement (about 0.92) is much higher than that before refinement (about 0.75), reflecting the retention efficiency of the core effective corpus; the proportions of noise / outlier corpus after refinement are both greatly reduced (noise from about 0.15 to nearly 0, outlier from about 0.10 to about 0.05), indicating that redundant / irrelevant information has been effectively filtered. The value of this type of chart is often used for project achievement report or academic analysis, which quantifies the effect of the corpus processing process through intuitive comparison of indicators, and verifies the practicability of the "quality evaluation and dynamic screening" link, which is a typical visualization way of data-driven process optimization.

[0047] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A policy knowledge graph-assisted multi-source domain corpus fusion method, characterized in that, The method includes the following steps: Obtain raw corpus sets from multiple independent domains, and perform deep structural feature analysis on the raw corpus sets of each domain to generate domain-structured corpus sets containing semantic units, entity labels, and association strength; The system receives the structured corpus set of the domain and performs a structural evaluation on it based on a pre-built policy knowledge graph. It identifies the mapping relationship and differences between corpus elements and graph nodes and edges, and generates graph alignment analysis results with mapping labels and difference measures. Extract the difference measure from the graph alignment analysis results and introduce real-time policy text stream for policy factor parsing. Embed the parsed policy impact factors into the corresponding nodes and edges of the knowledge graph to generate a policy-enhanced knowledge graph structure. Based on the policy-enhanced knowledge graph structure, heterogeneous normalization processing is performed on the multi-source domain structured corpus to generate a normalized corpus vector set; Perform multi-source collaborative alignment on the normalized corpus vector set, calculate the semantic similarity and relevance between vectors based on the semantic path of the knowledge graph, and generate a fused corpus cluster; Cross-domain consistency checks and conflict resolution are performed on the fused corpus cluster. The marked content is modified or merged according to predefined resolution rules to generate a consistent fused corpus.

2. The policy knowledge graph-assisted multi-source domain corpus fusion method according to claim 1, characterized in that, The specific steps for analyzing deep structural features include: The original corpus is segmented, part-of-speech tagging is performed, and named entity recognition is performed to extract basic semantic units and entity objects. Dependency parsing and semantic role labeling are performed on the identified semantic units and entity objects to construct a network of grammatical and logical relations within the sentence; Based on a pre-trained domain word vector model, the vector representations of each element in the semantic unit, entity object, and relation network, as well as the correlation strength scores between them, are calculated. The semantic units, entity objects, relation networks, vector representations, and association strength scores are integrated and encapsulated according to domain classification to form the domain-specific structured corpus.

3. The policy knowledge graph-assisted multi-source domain corpus fusion method according to claim 1, characterized in that, The process involves receiving the domain-structured corpus and performing a structural evaluation on it based on a pre-constructed policy knowledge graph. This evaluation identifies the mapping relationships and differences between corpus elements and graph nodes and edges, generating graph alignment analysis results with mapping labels and difference metrics. The entity objects and semantic relationships in the domain structured corpus are compared one by one with the nodes and edges in the policy knowledge graph; When an entity object or semantic relationship can be directly matched to a graph node or edge, it is recorded as a complete mapping, and the mapping path is marked. When an entity object or semantic relationship is semantically similar to, but not completely identical to, a graph node or edge, it is recorded as a fuzzy mapping, and the cosine distance in its vector space is calculated as a difference measure. When an entity object or semantic relationship is completely missing in the graph, it is recorded as unmapped and marked as an element to be expanded; All alignment results are summarized to form the map alignment analysis results, which include the fully mapped set, the fuzzy mapped set and their difference measure, and the unmapped set.

4. The policy knowledge graph-assisted multi-source domain corpus fusion method according to claim 1, characterized in that, The introduction of real-time policy text streams for policy factor analysis includes: Continuously crawl and monitor real-time policy text streams from designated official sources; The key policy clauses, subjects, objects, and constraints of the real-time policy text stream are extracted and structured. The structured policy provisions are associated and matched with the concept nodes in the policy knowledge graph to identify the graph regions directly affected by the policies. The impact intensity, direction of action, and timeliness of policies are transformed into calculable weighting factors, which are then embedded as attributes into the corresponding graph nodes and edges. The output is the policy-enhanced knowledge graph structure with embedded policy weight factors.

5. The policy knowledge graph-assisted multi-source domain corpus fusion method according to claim 1, characterized in that, The specific steps of the heterogeneous normalization process include: Define a unified metadata schema and semantic type system for the structured corpora of the domain from different sources; The original entity labels and relation types in the corpus are transformed into the unified semantic type system according to the mapping dictionary; The non-standardized vector representations of corpus elements are mapped to a unified standardized semantic vector space through a shared projection matrix; A unified metadata identifier containing source, processing timestamp, and version information is added to all corpus elements to generate a set of normalized corpus vectors in a standard format.

6. The policy knowledge graph-assisted multi-source domain corpus fusion method according to claim 1, characterized in that, The specific steps of the multi-source collaborative alignment operation include: Using the policy-enhanced knowledge graph structure as a semantic bridge, the shortest semantic path distance between any two corpus vectors in the normalized corpus vector set is calculated; Based on the phrase semantic path distance, construct a semantic similarity matrix and a semantic relevance matrix between corpus vectors; A density-based clustering algorithm is used to initially group the corpus vectors by combining the semantic similarity matrix and the semantic relevance matrix. Within each initial group, the corpus vectors are reordered and semantically reorganized based on semantic relevance to form the fused corpus cluster with a clear theme.

7. The policy knowledge graph-assisted multi-source domain corpus fusion method according to claim 1, characterized in that, The specific steps of the cross-domain consistency check and conflict resolution include: Within the fused corpus cluster, examine whether there are factual contradictions or numerical conflicts in the descriptions of the same entity or event from different source corpora; Check for unnecessary semantic overlap between different corpus fragments; For the identified factual contradictions, priority rules are set based on the authority, timeliness and fit of the corpus source and the knowledge graph. High-priority corpus is retained and low-priority corpus is marked or corrected. For the identified semantic overlap, a sentence boundary-based fusion algorithm is used for deduplication and smooth merging; The results of the verification and resolution are integrated into a logically consistent consistency fusion corpus.

8. The policy knowledge graph-assisted multi-source domain corpus fusion method according to claim 1, characterized in that, The method further includes: The policy-enhanced knowledge graph structure is iteratively updated based on the consistent fusion corpus. The fused new semantic relations and entities are injected into the graph, the connections between nodes are expanded and the weights of the edges are optimized, and an updated dynamic knowledge graph is generated. The updated dynamic knowledge graph is used to perform quality assessment and dynamic screening on the consistent fusion corpus, calculate the fit between corpus units and graph structure, filter low-quality or outlier corpus units based on fit threshold, and generate refined fusion corpus results. Establish a source tracing and feedback link between the refined fusion corpus results and the original multi-domain corpus, record the transformation history of each corpus unit during the fusion process, and dynamically calibrate the fusion rules based on the transformation history to generate a calibrated fusion rule set; Based on the calibrated fusion rule set and the updated dynamic knowledge graph.

9. The policy knowledge graph-assisted multi-source domain corpus fusion method according to claim 8, characterized in that, The iterative update of the policy-enhanced knowledge graph structure driven by the consistent fusion corpus includes: Newly emerging entities, attributes, and relationships between entities are extracted from the consistent fusion corpus. New entities are used as candidate nodes. Their semantic association with existing nodes in the graph is calculated. If the association exceeds a threshold, a connection is established; otherwise, a new node is created. New inter-entity relationships are used as candidate edges, and their weights are initialized using co-occurrence frequency and contextual strength from the corpus. The weights of existing edges in the graph are recalibrated using global statistical information from the consistency fusion corpus. Generate the updated dynamic knowledge graph, which includes new nodes, new edges, and updated weights.

10. The policy knowledge graph-assisted multi-source domain corpus fusion method according to claim 8, characterized in that, The specific steps of the quality assessment and dynamic screening include: Calculate the overall semantic fit between each corpus unit in the consistency fusion corpus and the updated dynamic knowledge graph. The fit is determined by the matching degree between entities within the unit and graph nodes, and the matching degree between relations and graph edges. A dynamic quality threshold is set, which is adjusted according to the goal of the fusion task and the overall distribution of the corpus; Speech units with semantic fit higher than the quality threshold are selected to form the core corpus set; The semantic fit of the corpus units is analyzed and classified as correctable noise corpus or irrelevant outlier corpus. The correctable noise corpus is then corrected under the guidance of the graph. The core corpus is merged with the revised corpus, and the output is the refined fused corpus result.

Images