Multi-source data intelligent treatment method for factual error correction

Abstract

The invention discloses a multi-source data intelligent treatment method for factual error correction. The method comprises the following steps: S1, multi-source data intelligent conflict resolution; s2, dynamically cleaning redundant data; s3, real-time data quality monitoring and early warning are carried out, and a multi-dimensional data quality evaluation index system including accuracy, integrity and timeliness is constructed; wherein the step S1, the step S2 and the step S3 form a collaborative closed loop, the cleaning result of the step S2 is fed back to the step S1 to optimize the weight parameter of the credibility evaluation model, and the abnormal data information of the step S3 is synchronized to the step S2 to update the screening rule of the redundancy graph. The method has the beneficial effects that according to the technical scheme, by quantifying the authority of the data source and the consistency of the data, a machine learning model is used for replacing manual conflict data screening, and the subjective deviation influence is avoided; automatic credible sorting of conflict data is realized, manual intervention links are reduced, and objectivity and efficiency of fact verification benchmark selection are improved.

Description

Technical Field

[0001] This invention relates to the fields of data governance and fact verification technology, and in particular to a multi-source data intelligent governance method for factual error correction. Background Technology

[0002] With the popularization of big data technology, factual error correction systems are increasingly used in fields such as academic paper verification and policy interpretation validation, relying on cross-validation of multi-source data. However, existing multi-source data governance technologies have significant drawbacks: Conflict resolution relies on manual intervention, which is inefficient and prone to bias: entity alignment is performed solely through knowledge graphs, without quantifying the authority and consistency of data sources. Conflicting data must be manually screened, which is susceptible to subjective judgment and thus inefficient.

[0003] The redundancy identification accuracy is insufficient and the processing flow is redundant: pure semantic recognition is used to remove redundancy without combining the authority of the data source to select the optimal version; traditional hash matching cannot identify redundant data with large differences in text expression but similar semantics, and all data needs to be processed repeatedly, consuming a lot of computing power.

[0004] Data quality monitoring is incomplete and lacks fault tolerance mechanisms: monitoring focuses on a single dimension and a multi-dimensional evaluation system covering accuracy, completeness, and timeliness has not been built; when the main data source is abnormal, there is no backup data source switching mechanism, which leads to the interruption of the fact verification system.

[0005] The lack of a closed-loop optimization mechanism makes it difficult to adapt to data changes: the focus on governance of a single aspect has failed to achieve a collaborative closed loop of conflict resolution, redundancy removal, and real-time monitoring, making it impossible to dynamically optimize governance effectiveness based on data feedback.

[0006] To address the aforementioned shortcomings, there is an urgent need for a multi-module collaborative intelligent governance method for multi-source data, which can improve the accuracy of factual error correction and system stability through quantitative assessment, precise semantic recognition, and real-time fault tolerance mechanisms. Summary of the Invention

[0007] The main technical problem solved by this invention is to provide a multi-source data intelligent governance method for factual error correction, thereby solving one or more of the problems mentioned above in the prior art.

[0008] To address the aforementioned technical problems, the present invention provides a technical solution: a multi-source data intelligent governance method for factual error correction, the innovation of which includes the following steps: S1: Intelligent conflict resolution of multi-source data. Integrate at least two types of heterogeneous data sources, construct a multi-dimensional data credibility assessment model based on machine learning, rank data with factual conflicts by credibility, and select highly authoritative and consistent data as the fact verification benchmark. S2: Dynamic cleaning of redundant data. Perform text clustering on the benchmark data selected in step S1 based on a semantic fingerprint algorithm to generate a data redundancy map. Retain the optimal data version and delete redundant copies according to the data source authority priority and data field integrity rules. Simultaneously, employ an incremental cleaning mechanism, comparing only the semantic fingerprints of existing data when new data is integrated, avoiding... Avoid full data duplication; S3: Real-time data quality monitoring and early warning, constructing a multi-dimensional data quality assessment index system including accuracy, completeness, and timeliness, and monitoring the update status of each data source in step S1 through real-time crawling. When data anomalies are detected, an early warning is automatically triggered, and a backup data source switching mechanism is started to ensure that the fact verification module continuously obtains usable data; Among them, steps S1, S2, and S3 form a collaborative closed loop. The cleaning results of step S2 are fed back to step S1 to optimize the weight parameters of the credibility assessment model, and the abnormal data information of step S3 is synchronized to step S2 to update the filtering rules of redundant maps.

[0009] In some implementations, the multi-dimensional data credibility assessment model in step S1 quantifies the authority index of the data source through a logistic regression algorithm and calculates the consistency index of the data source through a Transformer model. The authority index includes the qualification level of the institution to which the data source belongs, the data update frequency, and the verification accuracy of historical data. The consistency index is the degree of factual overlap between the target data and three or more other authoritative data sources. An overlap of ≥85% is considered as high consistency.

[0010] In some implementations, the fact conflict handling logic in step S1 includes: when the same fact differs between different authoritative data sources, the credibility assessment model assigns authoritative weights to different data sources, calculates a comprehensive credibility score by combining the consistency index, and selects the data from the higher authoritative data source as the verification benchmark when the score difference is ≥0.3.

[0011] In some implementations, the semantic fingerprint algorithm described in step S2 includes the following steps: The target data is segmented to extract core keywords; semantic vectors of the keywords are generated based on the BERT model, and the semantic vectors are compressed into 64-bit or 128-bit fingerprint sequences using the SimHash algorithm; the Hamming distance between fingerprint sequences of different data is calculated, and data with a distance of no more than 3 are judged as highly similar and included in the redundant map.

[0012] In some implementations, the specific process of the incremental cleaning mechanism described in step S2 is as follows: When new data is accessed, the semantic fingerprint algorithm steps are executed; the existing data fingerprint database in the Redis cache is called for comparison. If a fingerprint sequence with a Hamming distance of no more than 3 exists, redundancy determination is triggered; only data determined to be redundant is subjected to authority and integrity screening, while non-redundant data directly enters the fact verification module.

[0013] In some implementations, the quantitative standards for the multi-dimensional data quality assessment index system described in step S3 include: Accuracy: The error rate between data fields and authoritative benchmarks is no higher than 5%; Completeness: The missing rate of core key fields is no higher than 2%; Timeliness: The delay in dynamic data updates is no higher than 2 hours, and the delay in static data updates is no higher than 30 days.

[0014] In some implementations, the backup data source switching mechanism in step S3 includes: A backup data source library is pre-built, with priorities set according to backups from the same organization and authoritative alternatives in the same field. When the primary data source becomes unavailable, data is tampered with, or expires and is not updated, the system automatically calls the highest priority backup data source, with a switchover response time of ≤10 seconds. After the switchover is completed, an alert notification is sent to the administrator, and the data source weights in the credibility assessment model are updated.

[0015] In some implementations, abnormal data cases reported in step S3 are collected quarterly, and the multi-dimensional data credibility assessment model is updated through incremental training: the weight coefficients of the logistic regression model are updated, and the parameters of the top-level network layer of the Transformer model are fine-tuned.

[0016] The beneficial effects of this invention are: Enhancing the objectivity and efficiency of conflict resolution: By quantifying the authority and consistency of data sources, machine learning models replace manual screening of conflict data to avoid the influence of subjective bias; achieving automatic and reliable sorting of conflict data, reducing manual intervention, and improving the objectivity and efficiency of selecting fact verification benchmarks.

[0017] Accurately identify redundancy and optimize the processing flow: By integrating semantic fingerprinting technology with the authority and priority of data sources, it can accurately identify redundant data with large differences in text expression but similar semantics, and select and retain the best data version to avoid redundant data occupying storage resources; the incremental cleaning mechanism does not require repeated processing of the full amount of data, optimizes the data processing flow, and reduces the system's computing power consumption.

[0018] Ensuring continuous data availability and system stability: A multi-dimensional data quality assessment system covers key dimensions such as accuracy, completeness, and timeliness, enabling comprehensive monitoring of data quality status; a backup data source switching mechanism can quickly switch to an available backup data source when the primary data source is abnormal, preventing the fact verification system from stopping operation due to data interruption and ensuring continuous and stable system operation.

[0019] Lowering the barrier to cross-domain applications: By adapting domain-specific dictionaries and model weights, the method can be extended to multiple fields such as medicine, history, and policy without refactoring the system architecture, thus reducing the technical barriers and costs of cross-domain applications.

[0020] Achieving closed-loop dynamic optimization: The three modules of conflict resolution, redundancy cleaning, and real-time monitoring form a collaborative closed loop, which can continuously optimize model parameters and filtering rules based on abnormal cases fed back by monitoring, so that the data governance effect can be gradually improved with the application process and adapt to changes in data characteristics under different scenarios. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 This is a diagram of the overall architecture for intelligent governance of multi-source data.

[0022] Figure 2 is a flowchart of the conflict resolution module.

[0023] Figure 3 is a timing diagram of the backup data source switching. Detailed Implementation

[0024] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] like Figures 1 to 3 As shown, the embodiments of the present invention include: a multi-source data intelligent governance method for factual error correction, aiming to solve the pain points of existing technologies such as reliance on manual screening of conflicting data, low accuracy of redundant identification, high risk of data interruption, and lack of closed-loop optimization; the technical solution of the present invention will be described in detail below: I. Intelligent Conflict Resolution of Multi-Source Data 1. Data access and preprocessing: Access at least two types of heterogeneous data sources and achieve standardized processing through the following methods.

[0026] 1) Format standardization: Unstructured data is extracted using Python's PyPDF2 library to extract text. The library parses web pages and converts them into JSON format; structured data is directly synchronized to the system via API.

[0027] 2) Entity normalization: adopts The entity linking tool, combined with a domain-specific dictionary, maps synonymous entities to a unified identifier. For example, in the medical field, "chronic insomnia" and "primary insomnia" are uniformly mapped to ICSD-3 code G47.00 to avoid ambiguity in terminology.

[0028] 3) Conflict data identification: Conflicts are located through keyword matching and entity alignment. For example, by searching for "adult chronic insomnia and first-line treatment and treatment course", the differences in the description of treatment duration in different data sources can be identified.

[0029] 2. Construction and application of multi-dimensional data credibility assessment model: Construct a fusion model based on logistic regression and Transformer to calculate data credibility score.

[0030] 1) Authority score ( ): Quantified using a logistic regression model, the calculation formula is: in, (Institutional Qualification Weighting) (Update frequency weights) (Historical verification accuracy weighting); The ratings are as follows: government agencies 1.0, industry associations 0.7, commercial platforms 0.4, and personal forums 0.2. The data update frequencies are: daily 1.0, weekly 0.8, monthly 0.6, quarterly 0.4, and yearly 0.2. This represents the agreement rate between data from the data source and authoritative benchmarks over the past year.

[0031] 2) Consistency sub-score ( ): Calculated using the Transformer model, employing or The pre-trained model transforms the data text into a 768-dimensional semantic vector and calculates the mean cosine similarity with vectors from three or more authoritative data sources: Where n is the number of authoritative data sources selected. For the target data vector, For the data vector of the i-th authoritative data source, a high degree of consistency is determined if the overlap is ≥85%.

[0032] 3) Overall Credibility Score: The calculation formula is as follows: in, This is a weighting coefficient, ranging from 0.4 to 0.6, which can be dynamically adjusted according to the domain and optimized through gradient descent; for conflicting data, it is... Sort in descending order and select Data with a value of at least 0.8 and ranked first will be used as the fact verification benchmark; if the highest... If the value is less than 0.8, a manual review will be triggered.

[0033] 3. Fact Conflict Handling Logic: When the same fact differs across authoritative data sources, the credibility assessment model assigns corresponding authority weights to different data sources. For example, authoritative platforms have a weight of 0.7-0.8, while ordinary forums have a weight of 0.2-0.3. The overall credibility score is calculated by combining the consistency index. When the score difference is ≥0.3, the data from the more authoritative data source is selected as the verification benchmark.

[0034] II. Dynamic Cleaning of Redundant Data 1. Implementation of Semantic Fingerprint Algorithm 1) Core keyword extraction: using By combining domain-specific dictionaries, we extract the top 10 core keywords and focus on differentiated features.

[0035] 2) Semantic Vector and Fingerprint Generation: Based on the BERT model, the keyword sequence is transformed into a 512-dimensional semantic vector, through... The algorithm compresses the fingerprint into a 64-bit or 128-bit binary fingerprint: in, The weight of the k-th dimension vector is determined by the TF-IDF value of the keyword. The k-th dimension of the semantic vector is the numerical value. This is a sign function, with 1 for positive and 0 for negative.

[0036] 3) Redundancy determination: Calculate the Hamming distance between the fingerprint of the data to be detected and the existing fingerprint database. If the distance is not higher than 3, it is determined to be highly similar data and included in the redundancy map.

[0037] 2. Optimal version selection and incremental cleaning 1) Priority Calculation: Calculate the overall priority for redundant data: in, The highest priority data version is retained.

[0038] 2) Incremental cleaning mechanism: Build a Redis cache fingerprint database; when new data is accessed, only a new fingerprint is generated and compared with the Redis cache, without traversing the full data; non-redundant data directly enters the fact verification module, and redundant data is filtered for the optimal version and only one copy is retained.

[0039] III. Real-time Data Quality Monitoring and Early Warning 1. A multi-dimensional data quality assessment index system is constructed, which includes a quantitative index system that encompasses accuracy, completeness, and timeliness.

[0040] 1) Accuracy ( ): Require The error rate is no higher than 5%.

[0041] 2) Integrity ( ): Require The missing rate of core fields is no higher than 2%.

[0042] 3) Timeliness: Timeliness of dynamic data: ,Require The delay shall not exceed 2 hours; Timeliness of static data: ,Require The delay shall not exceed 30 days.

[0043] 2. Real-time monitoring and anomaly warning: A distributed crawler cluster based on the Scrapy framework is used, which can be configured with 10 nodes and the crawling frequency can be set according to the characteristics of the data source, such as once per hour for government websites and once every 2 hours for forums; the Flink real-time computing framework is used to calculate quality indicators for incoming data in real time; when data anomalies are detected.

[0044] 1) Data tampering detection: The SHA-256 hash value of the core data from the authoritative data source is stored in the consortium blockchain in advance. After crawling, the hash value is recalculated. If it is inconsistent, it is determined to be tampered.

[0045] 2) Alert Trigger: Send email alerts via the smtplib library and SMS alerts via the twilio API. The content includes the name of the abnormal data source, the type of abnormality, the current metric value, and the time when the abnormality occurred.

[0046] 3. Backup data source switching mechanism 1) Priority settings: Pre-build a backup data source library and set the priority according to the order of backup of the same institution > authoritative replacement of the same field.

[0047] 2) Switching trigger conditions: The switch will be automatically triggered when the main data source encounters a 404 error, data is tampered with, or the data is not updated within the specified time.

[0048] 3) Switching process: Call the backup data source API interface to synchronize core data, with a switch response time of ≤10 seconds; after the switch is completed, update the data source weights of the Redis fingerprint database and the trust evaluation model, and send a switch completion notification to the administrator.

[0049] IV. Module Collaboration and Model Optimization Closed Loop 1. Dynamic Feedback Optimization: The redundant cleaning results from step S2 are fed back to step S1 to optimize the credibility assessment model. Coefficients; abnormal data cases in step S3 are synchronized to step S2 to update the keyword weights of semantic fingerprints.

[0050] 2. Quarterly Incremental Training: Collect abnormal data cases from step S3 every quarter and update the model through incremental training.

[0051] 1) Logistic Regression Model: Using SGDClassifier from the scikit-learn library, with a learning rate of 0.001, 200 iterations, and updates... , , .

[0052] 2) Transformer model: Freeze the bottom 10 layers of BERT, fine-tune the top 2 layers, train batch size 32, optimizer used AdamW, weight decay 0.01.

[0053] The advantages of this technical solution are: Enhancing the objectivity and efficiency of conflict resolution: By quantifying the authority and consistency of data sources, machine learning models replace manual screening of conflict data to avoid the influence of subjective bias; achieving automatic and reliable sorting of conflict data, reducing manual intervention, and improving the objectivity and efficiency of selecting fact verification benchmarks.

[0054] Accurately identify redundancy and optimize the processing flow: By integrating semantic fingerprinting technology with the authority and priority of data sources, it can accurately identify redundant data with large differences in text expression but similar semantics, and select and retain the best data version to avoid redundant data occupying storage resources; the incremental cleaning mechanism does not require repeated processing of the full amount of data, optimizes the data processing flow, and reduces the system's computing power consumption.

[0055] Ensuring continuous data availability and system stability: A multi-dimensional data quality assessment system covers key dimensions such as accuracy, completeness, and timeliness, enabling comprehensive monitoring of data quality status; a backup data source switching mechanism can quickly switch to an available backup data source when the primary data source is abnormal, preventing the fact verification system from stopping operation due to data interruption and ensuring continuous and stable system operation.

[0056] Lowering the barrier to cross-domain applications: By adapting domain-specific dictionaries and model weights, the method can be extended to multiple fields such as medicine, history, and policy without refactoring the system architecture, thus reducing the technical barriers and costs of cross-domain applications.

[0057] Achieving closed-loop dynamic optimization: The three modules of conflict resolution, redundancy cleaning, and real-time monitoring form a collaborative closed loop, which can continuously optimize model parameters and filtering rules based on abnormal cases fed back by monitoring, so that the data governance effect can be gradually improved with the application process and adapt to changes in data characteristics under different scenarios.

[0058] Example 1 This embodiment focuses on the factual verification of first-line treatment regimens for adult chronic insomnia (meeting ICSD-3 diagnostic criteria). It applies the aforementioned technical solution to resolve issues such as conflicting treatment courses, redundant guideline versions, and delayed data updates in multi-source data. The specific process is as follows: (I) Intelligent Conflict Resolution of Multi-Source Data 1. Data Source Access and Preprocessing 1) Select four types of heterogeneous data sources, categorized according to their authority: Data source type Specific source Authoritative indicator parameters Authoritative medical database UpToDate (Sleep Medicine Topic) Institutional Qualification = 0.9 (International authoritative clinical decision database), Update Frequency = Quarterly (0.9), Historical Verification Accuracy = 97% (0.97) Industry Guide Platform Guidelines for the Diagnosis and Treatment of Insomnia in Chinese Adults (2024 Edition) issued by the Sleep Disorders Group of the Neurology Branch of the Chinese Medical Association. Institutional Qualification = 0.8 (National Academic Society), Update Frequency = Annual Update (0.8), Historical Verification Accuracy = 95% (0.95) General Health Forum DXY.com Forum "Sleep Health Section" Institutional Qualification = 0.4 (Commercial Health Platform), Update Frequency = Weekly (0.6), Historical Verification Accuracy = 82% (0.82) Authoritative benchmark source World Sleep Society (WFSRS) Global Consensus on the Management of Chronic Insomnia (2025 Edition) Institutional Qualifications = 1.0 (International Industry Authority), Update Frequency = Biennial (Updated every two years, 0.7), Historical Verification Accuracy = 100% (1.0) 2) Preprocessing operations: Format conversion: UpToDate data was converted to JSON using the Python lxml library to extract core fields; the main text of the "China Guidelines (2024 Edition)" was extracted and the core content was located using PyMuPDF; data from the DXY.cn forum was used to remove irrelevant replies using BeautifulSoup4, retaining doctor replies and guideline citations.

[0059] Entity normalization: using spaCy 3.6.0 to load the en_med7_large model and a custom sleep domain dictionary, synonymous disease terms are uniformly mapped to ICSD-3 encoding G47.00.

[0060] Conflict identification: After searching the keywords, the core conflict was found. UpToDate records "CBT-I standard course of treatment is 8 weeks, once a week, 45-60 minutes each time"; a grassroots doctor on the DXY forum replied that "CBT-I course of treatment is 6 weeks, twice a week, 30 minutes each time"; the "China Guidelines (2024 Edition)" does not specify the course of treatment.

[0061] 2. Credibility assessment model calculation and parameter configuration: Logistic regression model weights =0.5、 =0.3、 =0.2, domain fit α=0.6; the Transformer model uses a biomedical pre-trained model, and the input text length is set to 512 tokens.

[0062] Credibility scores for each data source: 1) UpToDate: , =0.94, Overall Score ; 2) DXY.cn (Dingxiang Doctor Forum): , =0.68, Overall Score ; 3) China Guide (2024 Edition): , =0.92, Overall Score .

[0063] Benchmark filtering results: Select UpToDate data As a factual verification benchmark, the specific technical components of CBT-I in the "China Guide (2024 Edition)" are also added as supplementary benchmark data.

[0064] (ii) Dynamic cleaning of redundant data 1. Semantic fingerprint generation and redundancy determination: The top 5 core keywords were extracted from the UpToDate benchmark data: adult chronic insomnia, CBT-I, 8-week course of treatment, once a week, 45-60 minutes / session; a weighted semantic vector was generated by the BERT model, with a weight of 0.4 for course duration, 0.3 for frequency, and 0.3 for single session duration, and compressed into a 64-bit binary fingerprint by the SimHash algorithm.

[0065] The system newly integrated the "Chronic Insomnia Diagnosis and Treatment Pathway" from the official website of a top-tier hospital's sleep medicine center. After generating a semantic fingerprint, it was compared with the Redis cache. The Hamming distance was 2, indicating that the data was redundant. The system recorded an 8-week CBT-I treatment course, once a week, for 40-60 minutes each time.

[0066] 2. Optimal version selection priority: Hospital official website and institutional qualifications 0.7 ( =0.7), core field missing rate 20% ( =0.8), priority score Benchmark data priority score The UpToDate baseline data is retained, redundant copies of the hospital's official website are deleted, and the details of "single session duration of 40-60 minutes" in the hospital data are stored in the associated database as a baseline supplementary description.

[0067] (III) Real-time data quality monitoring and early warning 1. Monitoring configuration and quality assessment: A 3-node distributed crawler cluster is adopted, and the crawling frequency is set according to the data source update cycle: UpToDate once every 15 days, "China Guide (2024 Edition)" once a month, DXY.cn forum once every 3 days, and WFSRS consensus once every 3 months.

[0068] Calculation results of multi-dimensional quality indicators: 2. Exception handling and backup switchover: Simulate the WFSRS official website going offline due to server maintenance, resulting in the crawler failing 5 consecutive requests and the backup mirror site being inaccessible.

[0069] 1) Warning Trigger: Send an email to the administrator via smtplib with the subject "WFSRS Authoritative Source Anomaly" and the content including the anomaly type, occurrence time, and scope of impact; at the same time, push the warning to the technical group via the WeChat API.

[0070] 2) Backup switch: The WFSRS consensus backup library of the National Medical Library is called according to priority, and the 2025 version consensus data is synchronized through the FTP interface; after the switch is successful, the WFSRS data fingerprint in the Redis fingerprint library is updated, and the fact verification system is not interrupted.

[0071] (iv) Module Collaboration Optimization 1. Collection of abnormal cases: Collect three types of abnormal cases within one quarter: misjudgment of conflicts, missed detection of redundancy, and monitoring lag.

[0072] 2. Model and rule optimization: 1) Credibility Assessment Model: The logistic regression model now includes a new "Target Audience" indicator with a weight of 0.1. =0.5、 =0.3、 =0.1、 =0.1; The Transformer model adds sleep domain labels to the input and fine-tunes the top 3 layers.

[0073] 2) Cleaning and monitoring rules: "Additional treatment plan" has been added as an exclusion item for redundancy judgment in the redundancy cleaning; the data crawling frequency of local hospital sleep specialists has been adjusted from once every 7 days to once every 3 days.

[0074] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A multi-source data intelligent governance method for factual error correction, characterized in that: Includes the following steps: S1: Intelligent conflict resolution of multi-source data, accessing at least two types of heterogeneous data sources, constructing a multi-dimensional data credibility assessment model based on machine learning, ranking data with factual conflicts by credibility, and selecting data with high authority and high consistency as the fact verification benchmark; S2: Dynamic cleaning of redundant data. Based on the semantic fingerprint algorithm, text clustering is performed on the baseline data filtered in step S1 to generate a data redundancy map. The optimal data version is retained and redundant copies are deleted according to the authority priority of the data source and the data field integrity rules. At the same time, an incremental cleaning mechanism is adopted. When new data is introduced, only the semantic fingerprint of the existing data is compared to avoid full data duplication. S3: Real-time data quality monitoring and early warning. A multi-dimensional data quality evaluation index system including accuracy, completeness and timeliness is constructed. The update status of each data source in step S1 is monitored in real time by crawling. When data anomalies are detected, an early warning is automatically triggered and a backup data source switching mechanism is started to ensure that the fact verification module continuously obtains usable data. Among them, steps S1, S2 and S3 form a collaborative closed loop. The cleaning results of step S2 are fed back to step S1 to optimize the weight parameters of the credibility evaluation model. The abnormal data information of step S3 is synchronized to step S2 to update the filtering rules of the redundant map.

2. The multi-source data intelligent governance method for factual error correction according to claim 1, characterized in that: The multi-dimensional data credibility assessment model in step S1 quantifies the authority index of the data source through logistic regression algorithm and calculates the consistency index of the data source through Transformer model. The authority index includes the qualification level of the institution to which the data source belongs, the data update frequency, and the verification accuracy of historical data. The consistency index is the degree of factual overlap between the target data and three or more other authoritative data sources. An overlap of ≥85% is considered as high consistency.

3. The multi-source data intelligent governance method for factual error correction according to claim 1, characterized in that: The logic for handling factual conflicts in step S1 includes: when the same fact differs between different authoritative data sources, the credibility assessment model assigns authoritative weights to different data sources, calculates a comprehensive credibility score by combining the consistency index, and selects the data from the higher authoritative data source as the verification benchmark when the score difference is ≥0.

3.

4. The multi-source data intelligent governance method for factual error correction according to claim 1, characterized in that: The implementation process of the semantic fingerprint algorithm in step S2 includes: The target data is segmented to extract core keywords; semantic vectors of the keywords are generated based on the BERT model, and the semantic vectors are compressed into 64-bit or 128-bit fingerprint sequences using the SimHash algorithm; the Hamming distance between fingerprint sequences of different data is calculated, and data with a distance of no more than 3 are judged as highly similar and included in the redundant map.

5. The multi-source data intelligent governance method for factual error correction according to claim 1, characterized in that: The specific process of the incremental cleaning mechanism described in step S2 is as follows: When new data is accessed, the semantic fingerprint algorithm steps are executed; the existing data fingerprint database in the Redis cache is called for comparison. If a fingerprint sequence with a Hamming distance of no more than 3 exists, redundancy determination is triggered; only data determined to be redundant is subjected to authority and integrity screening, while non-redundant data directly enters the fact verification module.

6. The multi-source data intelligent governance method for factual error correction according to claim 1, characterized in that: The quantitative standards for the multi-dimensional data quality assessment index system mentioned in step S3 include: Accuracy: The error rate between data fields and authoritative benchmarks is no higher than 5%; Completeness: The missing rate of core key fields is no higher than 2%; Timeliness: The delay in dynamic data updates is no higher than 2 hours, and the delay in static data updates is no higher than 30 days.

7. The multi-source data intelligent governance method for factual error correction according to claim 1, characterized in that: The backup data source switching mechanism mentioned in step S3 includes: A backup data source library is pre-built, with priorities set according to backups from the same organization and authoritative alternatives in the same field. When the primary data source becomes unavailable, data is tampered with, or expires and is not updated, the system automatically calls the highest priority backup data source, with a switchover response time of ≤10 seconds. After the switchover is completed, an alert notification is sent to the administrator, and the data source weights in the credibility assessment model are updated.

8. The multi-source data intelligent governance method for factual error correction according to claim 1, characterized in that: Quarterly, abnormal data cases reported in step S3 are collected, and the multi-dimensional data credibility assessment model is updated through incremental training: the weight coefficients of the logistic regression model are updated, and the parameters of the top-level network layer of the Transformer model are fine-tuned.

Images