Multi-source heterogeneous data verification and tracing method and device and related equipment

Abstract

The invention relates to the technical field of data processing, can be applied to the field of science and technology finance / digital medical treatment, and discloses a multi-source heterogeneous data verification and tracing method and device and related equipment. The method comprises the following steps: acquiring multi-source heterogeneous data from a service data source of a target service; preprocessing the multi-source heterogeneous data, extracting a first data field in each data source, and matching the first data fields with the same semantics in different data sources based on a preset semantic mapping rule; performing consistency verification on the matched first data fields to obtain second data fields with inconsistent verification results; performing root cause tracing on the second data field based on a preset difference analysis rule, and determining a difference type corresponding to the second data field according to a root cause tracing result; and generating a multi-source heterogeneous data verification and tracing result according to the business data source, the second data field, the root cause tracing result and the difference type. According to the method, matching, difference identification and root cause positioning of multi-source heterogeneous data are realized.

Description

Technical Field

[0001] This application relates to the field of data processing technology and can be applied to the fields of financial technology / digital healthcare, and in particular to a method, apparatus and related equipment for multi-source heterogeneous data verification and traceability. Background Technology

[0002] As data becomes increasingly important in various application areas, its accuracy, consistency, and integrity directly impact business decisions and operations. Currently, data verification and discrepancy tracking technologies primarily rely on structured data comparison tools (such as Beyond Compare) and file parsing tools (such as Apache PDFBox) to achieve basic data verification functions, meeting the needs of some simple scenarios. However, existing technologies have significant limitations when processing multi-source heterogeneous data: First, they lack multi-format compatibility, making it difficult to achieve unified verification across data types. For example, in the financial sector, insurance companies need to process PDF policy files, Excel premium statistics tables, and customer information in databases simultaneously. Existing tools have limited ability to extract key information from unstructured PDF files and cannot intelligently correlate and compare data in PDFs with corresponding fields in databases. Second, they lack semantic-level intelligent matching capabilities, relying on static mapping rules. For example, in the medical field, different systems may describe "patient gender" as "male / female" or "M / F." Existing tools struggle to automatically match such semantically equivalent fields, requiring manual intervention to set mapping relationships. Third, their discrepancy tracking capabilities are weak, only able to mark surface differences without analyzing root causes. For example, when the insurance industry discovers discrepancies in policy amounts, existing tools cannot distinguish whether the discrepancy is due to a unit conversion error (e.g., "yuan" versus "ten thousand yuan") or a data entry mistake. Similarly, when a deviation in test results is detected in a medical system, it is difficult to pinpoint whether the issue stems from the precision of laboratory equipment or distortion during data transmission, making problem rectification challenging. Summary of the Invention

[0003] The main technical problem addressed by the embodiments of this application is the inadequacy of traditional data processing techniques in terms of compatibility and dynamic matching when processing multi-source heterogeneous data.

[0004] To address the aforementioned technical problems, the first technical solution adopted in this application is: providing a method for verifying and tracing multi-source heterogeneous data, comprising: obtaining multi-source heterogeneous data in a preset data format from a business data source of a preset target business, wherein the multi-source heterogeneous data contains at least two different data formats of business data; preprocessing the multi-source heterogeneous data, extracting a first data field from each data source, and matching the first data fields with the same semantic meaning in different data sources based on a preset semantic mapping rule; performing consistency verification on the matched first data fields to obtain a second data field whose verification result is inconsistent; performing root cause tracing on the second data field based on a preset difference analysis rule, and determining the difference type corresponding to the second data field based on the root cause tracing result; and generating a multi-source heterogeneous data verification and tracing result based on the business data source, the second data field, the root cause tracing result, and the difference type.

[0005] Optionally, the preset data format includes portable document format, spreadsheet format, and database table format. The step of preprocessing the multi-source heterogeneous data and extracting the first data field from each data source includes: extracting business information from the multi-source heterogeneous data in the portable document format using optical character recognition to obtain the first data field scattered across different pages; performing table structure parsing on the multi-source heterogeneous data in the spreadsheet format using a preset table data parsing tool to obtain the first data field in the cell; and performing field mapping transformation on the multi-source heterogeneous data in the database table format based on a preset mapping relationship table between database fields and business fields, and performing data type standardization long distance to obtain the standardized first data field.

[0006] Optionally, the step of performing consistency verification on the matched first data field to obtain a second data field with inconsistent verification results includes: comparing the same semantic key value matched and confirmed in different portable documents field by field to verify whether the key values ​​corresponding to the key name expression variants are consistent, and obtaining a first verification result; if the portable document contains cross-page distribution information, associating and splicing the key value information of the same business object scattered on different pages, and then performing consistency verification on the associated key value information to obtain a second verification result; comparing the first data field extracted from the portable document with the same semantic field matched in non-portable documents to obtain a third verification result; analyzing the first verification result, the second verification result, and the third verification result, if there is a key value mismatch or the key value deviation exceeds a preset threshold range, then marking the corresponding first data field as the second data field.

[0007] Optionally, the step of performing consistency verification on the matched first data field to obtain a second data field with inconsistent verification results includes: extracting the first data field from a hybrid format data source composed of portable document format and tabular format, and generating a corresponding business dataset according to the business logic of the preset target business; comparing the same semantic data field in the business dataset and the target portable document item by item through an automated verification script to verify the correspondence and numerical consistency between the associated data items and the target declared data items, and obtaining a fourth verification result; performing cross-source composite verification on the associated data in the hybrid format data source and the corresponding business data in the hybrid format data source composed of portable document format and tabular format, and obtaining a fifth verification result; analyzing the fourth verification result and the fifth verification result, and if there is a mismatch in the business data association logic, or the deviation of the associated field combination result exceeds a preset threshold range, then marking the corresponding first data field as the second data field.

[0008] Optionally, the step of performing consistency verification on the matched first data field to obtain a second data field with inconsistent verification results includes: performing field-level association mapping between fields in the database table and row and column data in the spreadsheet file based on preset field mapping rules; performing item-by-item verification on the mapped fields to obtain a sixth verification result, wherein the item-by-item verification includes automatic data type conversion and adaptation, and numerical deviation verification based on a preset fault tolerance threshold; capturing dynamic business data in the system interface or logs in real time, and performing automatic matching verification with preset benchmark data in the offline spreadsheet template to obtain a seventh verification result; analyzing the sixth verification result and the seventh verification result, and if there is a field mapping mismatch, a numerical deviation exceeding the fault tolerance threshold, or dynamic data not synchronized with the benchmark data, then marking the corresponding first data field as the second data field.

[0009] Optionally, the step of tracing the root cause of the second data field based on preset difference analysis rules includes: locating the root cause of the second data field in preset dimensions using a rule engine and semantic reasoning based on preset data verification rules for the target business; obtaining multi-version business files and historical database snapshots associated with the second data field through a data lineage analysis mechanism, and constructing a data flow link diagram based on the multi-version business files and historical database snapshots; performing a traceability comparison between the second data field and the corresponding field in the historical version data, and locating the difference-generating node in the data flow link diagram based on the traceability comparison results; determining the difference cause type based on the data operation logs, data processing scripts, and data permission records of the difference-generating node using a preset causal association analysis rule set; generating a traceability analysis conclusion containing a root cause description, a responsible node identifier, and improvement suggestions based on the difference cause type, and associating the traceability analysis conclusion with the corresponding difference-generating node.

[0010] Optionally, the step of generating multi-source heterogeneous data verification and tracing results based on the business data source, the second data field, the root cause tracing result, and the difference type includes: generating initial verification and tracing results according to a preset structured report template, based on the source identifier of the business data source, the field content and location information of the second data field, the difference generation nodes and flow link segments in the root cause tracing result, and the classification statistics of the difference type; visualizing the initial verification and tracing results, the visualization including a difference heatmap to display the inconsistency distribution density of multi-source data, highlighting the difference generation links in the link nodes; adding an interactively displayable data flow link diagram and a comparison record of historical data versions to the initial verification and tracing results to obtain the multi-source heterogeneous data verification and tracing results.

[0011] To address the aforementioned technical problems, the second technical solution adopted in this application is: providing a multi-source heterogeneous data verification and traceability device, comprising: a multi-source heterogeneous data acquisition module, used to acquire multi-source heterogeneous data in a preset data format from a business data source of a preset target business, wherein the multi-source heterogeneous data contains at least two different data formats of business data; a multi-source heterogeneous data preprocessing module, used to preprocess the multi-source heterogeneous data, extract a first data field from each data source, and match the first data fields with the same semantic meaning in different data sources based on a preset semantic mapping rule; a data consistency verification module, used to perform consistency verification on the matched first data fields to obtain a second data field whose verification result is inconsistent; a data root cause tracing module, used to perform root cause tracing on the second data field based on a preset difference analysis rule, and determine the difference type corresponding to the second data field according to the root cause tracing result; and a verification and tracing result module, used to generate multi-source heterogeneous data verification and tracing results based on the business data source, the second data field, the root cause tracing result, and the difference type.

[0012] To solve the above-mentioned technical problems, the third technical solution adopted in the embodiments of this application is: to provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-source heterogeneous data verification and traceability method as described above.

[0013] To solve the above-mentioned technical problems, the fourth technical solution adopted in the embodiments of this application is: to provide a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by an electronic device, the electronic device performs the multi-source heterogeneous data verification and traceability method as described above.

[0014] Unlike related technologies, this application achieves cross-source semantic field matching by acquiring multi-source heterogeneous data and preprocessing it, thus overcoming format and semantic limitations; it accurately identifies inconsistent data through consistency verification, improving the comprehensiveness of detection; it traces the root cause based on rules and determines the type of difference, providing a clear direction for data quality rectification; and the final generated results integrate key information, providing technical support for multi-domain data management, data compliance review, and data quality optimization. Attached Figure Description

[0015] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0016] Figure 1 This is a schematic diagram of the operating environment of the multi-source heterogeneous data verification and traceability method provided in the embodiments of this application.

[0017] Figure 2 This is a schematic diagram of the execution flow of the multi-source heterogeneous data verification and traceability method provided in the embodiments of this application.

[0018] Figure 3 This is a schematic diagram of the execution flow of consistency verification in the multi-source heterogeneous data verification and traceability method provided in the embodiments of this application.

[0019] Figure 4 This is a schematic diagram of the execution flow of root cause tracing in the multi-source heterogeneous data verification and tracing method provided in the embodiments of this application.

[0020] Figure 5 This is a schematic diagram of the system structure of the multi-source heterogeneous data verification and traceability device provided in the embodiments of this application.

[0021] Figure 6 This is a schematic diagram of the hardware structure of an electronic device that performs a multi-source heterogeneous data verification and traceability method according to an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Software tools, components, or servers not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0023] It should be noted that, unless otherwise specified, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device schematic diagram or the order in the flowchart.

[0024] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0025] To facilitate understanding of this embodiment, a detailed description of a multi-source heterogeneous data verification and traceability method disclosed in this application embodiment will be provided first. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the operating environment of the multi-source heterogeneous data verification and traceability method provided in the embodiments of this application, such as... Figure 1 As shown, the execution subject of the multi-source heterogeneous data verification and traceability method provided in this application embodiment is generally an electronic device with a certain computing power, such as a computer device. In some possible implementations, this multi-source heterogeneous data verification and traceability method can be implemented by the processor calling computer-readable instructions stored in the memory. Figure 1 The computer equipment mentioned can be a server. A server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This can be understood as... Figure 1 The number of computer devices shown is merely illustrative and can be expanded in any number according to actual needs.

[0026] Please continue reading. Figure 2 , Figure 2 This is a schematic diagram of the execution flow of the multi-source heterogeneous data verification and traceability method provided in the embodiments of this application, such as... Figure 2 As shown, it includes the following steps:

[0027] S1. Obtain multi-source heterogeneous data in a preset data format from the business data source of the preset target business. The multi-source heterogeneous data contains at least two different data formats of business data.

[0028] Step S1, as the initial stage of the multi-source heterogeneous data verification and traceability process, focuses on establishing a standardized data input mechanism. Firstly, it involves the precise binding of pre-defined target businesses with business data sources. By pre-defining specific business scenarios (e.g., insurance underwriting, medical record management), the scope of data acquisition is locked to data sources strongly related to that business, ensuring the business relevance of the data. For example, in the insurance scenario, pre-defined data sources include the policy database of the underwriting system, PDF application documents submitted by agents, and Excel premium ledgers from the finance department; in the medical scenario, this includes patient information tables from the hospital's HIS system, the LIS database of the laboratory department, and PDF documents generated from scanned handwritten medical records. Secondly, it requires pre-configuring the technical coverage capabilities of pre-defined data formats. This necessitates pre-configuring support for parsing structured data (e.g., MySQL tables in the insurance policy database, Oracle tables for hospital patient information), semi-structured data (e.g., Excel details of insurance premiums, CSV reports of medical examination items), and unstructured data (e.g., PDF documents of insurance terms, PDF medical imaging reports) to avoid data loss due to format incompatibility. Finally, in "multi-source heterogeneous," "multi-source" refers to the distributed nature of data sources (for example, insurance data may come from headquarters systems, branch office terminals, and third-party platforms; medical data may come from outpatient clinics, inpatient facilities, and physical examination centers), while "heterogeneous" refers to the technical differences in storage formats. For instance, in the insurance field, PDF insurance application documents require OCR to extract text, Excel ledgers require parsing formula dependencies, and database tables require reading field constraints; in the medical field, PDF medical records require recognizing paragraph semantics, and database tables require extracting field relationships. Step S1 above reduces interference from invalid data through a pre-set mechanism, and at the same time, its multi-format compatibility covers the diverse forms of business data, laying a technical foundation for subsequent field matching and verification.

[0029] S2. Preprocess multi-source heterogeneous data, extract the first data field from each data source, and match the first data field with the same semantic meaning in different data sources based on preset semantic mapping rules.

[0030] For example, key information can be extracted from data in different formats first, and then cross-source correspondence can be established through semantic association. In the insurance scenario, it is necessary to extract "policyholder's name" from PDF insurance application forms, "insured's name" from Excel premium tables, and "insured_name" from policy databases. Through preset semantic mapping rules (such as the semantic equivalence of "policyholder" and "insured" in individual policies, and the correspondence between Chinese names and English fields), the aforementioned semantically identical fields can be matched into comparable datasets. In the medical scenario, the "patient's age" in PDF medical records, the "age" in Excel physical examination forms, and the "age" field in the database can be semantically aligned.

[0031] As an optional implementation, the preset data format mentioned above includes portable document format (PDF format), spreadsheet format (such as Excel format) and database table format. The step of preprocessing multi-source heterogeneous data and extracting the first data field from each data source in step S2 can specifically include the following steps S21 to S23.

[0032] S21. Business information is extracted from multi-source heterogeneous data in portable document format using optical character recognition to obtain the first data field scattered across different pages.

[0033] Among these, processing portable document format data addresses the problem of extracting structured information from unstructured data. Optical character recognition (OCR) technology converts handwritten or printed text in scanned PDFs into editable characters. This is then combined with layout analysis algorithms to locate key information. For example, in the insurance field, scattered fields such as "insurance amount" and "payment period" are extracted from different pages of a PDF insurance application; in the medical field, fragmented information such as "chief complaint" and "past medical history" is extracted from different chapters of a PDF medical record, ultimately forming the first data field that can be used for subsequent matching.

[0034] S22. Use the preset table data parsing tool to perform table structure parsing on the multi-source heterogeneous data in spreadsheet format to obtain the first data field in the cell.

[0035] The system utilizes a pre-defined table parsing tool to identify table headers, row and column relationships, and cell data types, eliminating interference from merged cells, blank rows, and other factors. For example, in an insurance scenario, it extracts column data such as "policy number" and "premium receivable" from an Excel premium details table, while simultaneously processing the formula calculation results within the cells; in a medical scenario, it extracts structured fields such as "test items," "result values," and "reference ranges" from an Excel test report, ensuring that the extracted primary data fields are consistent with the table's logic.

[0036] S23. Based on the preset mapping relationship table between database fields and business fields, perform field mapping transformation on the multi-source heterogeneous data in the database table format, and perform data type standardization long distance to obtain the standardized first data field.

[0037] For example, based on a pre-defined mapping table (e.g., the database field "pol_no" corresponds to the business field "policy number" and "pt_age" corresponds to "patient age"), technically named fields are converted into business-understandable names; at the same time, data types are standardized, for example, the date format in the insurance database is unified to "YYYY-MM-DD" and the numerical units in the medical database are unified to international standard units, and finally the standardized first data field is output.

[0038] Through steps S21 to S23, unstructured information in portable document formats is transformed into extractable fields, solving the problem of scattered PDF data; effective information is accurately extracted from spreadsheet formats, avoiding field loss caused by formatting issues; and technical fields in database tables are converted into standardized business fields, eliminating naming and type differences. The final output of a unified format first data field provides a consistent foundation for cross-source semantic matching, significantly improving the integration efficiency and matching accuracy of multi-source heterogeneous data.

[0039] S3. Perform a consistency check on the first data field after matching, and obtain the second data field that is inconsistent with the check result.

[0040] As an alternative implementation method, please continue reading. Figure 3 , Figure 3 This is a schematic diagram illustrating the execution flow of consistency verification in the multi-source heterogeneous data verification and traceability method provided in this application embodiment, as shown below. Figure 3 As shown, step S3 above can specifically include steps S31 to S34.

[0041] S31. Compare each field of the same semantic key value that has been matched and confirmed in different portable documents to verify whether the key values ​​corresponding to the key name expression variants are consistent, and obtain the first verification result.

[0042] For example, in an insurance scenario, the "insurance amount" in PDF document A and the "insurance amount" in PDF document B are variants of key name representation. The accuracy of the data needs to be verified by comparing whether the corresponding values ​​of the two are consistent. In a medical scenario, comparing the key values ​​of "patient height" and "height" in different PDF medical records can identify differences in the recording of the same patient information in different documents, and finally form the first verification result.

[0043] S32. If the portable document contains cross-page distribution information, associate and concatenate the key-value information of the same business object scattered across different pages, and then perform consistency verification on the associated key-value information to obtain the second verification result.

[0044] When portable documents contain information distributed across multiple pages, it is necessary to first establish the association relationship of the same business object before verification. Technically, this is achieved by identifying unique identifiers within the document (e.g., the "policy number" in an insurance PDF, the "medical record number" in a medical PDF), and associating and splicing key-value information scattered across different pages (e.g., the "policyholder information" and "beneficiary information" located on the first and last pages of an insurance policy, and the "examination date" and "examination result" on different pages of a medical record) to form a complete business data chain before verifying consistency. This solves the problem of incomplete verification caused by information fragmentation and yields a second verification result.

[0045] S33. Perform a cross-format comparison between the first data field extracted from the portable document and the same semantic field matched in the non-portable document to obtain the third verification result.

[0046] This involves cross-format comparison of fields extracted from portable documents with semantically identical fields from non-portable documents to eliminate consistency discrepancies caused by different data carriers. For example, in an insurance scenario, the "premium amount" in a PDF insurance application needs to be compared with the "actual amount received" in an Excel premium table and the "payment_amount" in the database; in a medical scenario, the "blood glucose value" in a PDF diagnostic report needs to be verified for accuracy with the "blood glucose test result" in an Excel test table and the "blood_glucose" in the database. Cross-format verification ensures the accuracy of data during the transfer process, forming a third verification result.

[0047] S34. Analyze the first verification result, the second verification result, and the third verification result. If there is a key-value mismatch or the key-value deviation exceeds the preset threshold range, mark the corresponding first data field as the second data field.

[0048] By comprehensively analyzing the results of the first three verification steps, inconsistent fields can be accurately located. Technically, clear judgment rules are established: when key-value pairs do not match at all, or when the deviation of a numerical field exceeds a preset threshold (e.g., an insurance amount error exceeding 0.1%, or a medical test value exceeding the normal fluctuation range), the corresponding first data field is marked as the second data field. This step achieves structured output of the verification results through quantitative standards, providing a clear target object for subsequent root cause tracing.

[0049] As another optional implementation, step S3 may further include steps S351 to S354.

[0050] S351. Extract the first data field from the mixed format data source composed of portable document format and tabular format, and generate the corresponding business dataset according to the business logic of the preset target business.

[0051] For example, in the insurance scenario, the "insured information" of the PDF insurance application form is linked with the "premium calculation items" of the Excel detailed table to form a complete business dataset containing the insured entity and cost information; in the medical scenario, the "diagnosis conclusion" of the PDF medical record is linked with the "test indicators" of the Excel examination form by patient ID to build a complete diagnosis and treatment data chain, providing a structured data foundation for subsequent verification.

[0052] S352. Using an automated verification script, the same semantic data field in the business dataset and the target portable document is compared item by item to verify the correspondence between the associated data items and the target declared data items and the consistency of the values, thus obtaining the fourth verification result.

[0053] For example, in an insurance scenario, a script is used to compare the associated "basic policy information + premium calculation results" dataset with the "insurance liability" and "premium payable" fields in the target PDF policy to check whether the correspondence is accurate and whether the values ​​match; in a medical scenario, a script is used to verify the "diagnosis conclusion + test indicators" dataset with the "final diagnosis" and "reference value" fields in the target PDF medical record to ensure that the data association logic and values ​​are without deviation, and finally obtain the fourth verification result.

[0054] S353. Perform cross-source composite verification on the associated data in the mixed format data source and the corresponding business data in the mixed format data source composed of portable document format and tabular format to obtain the fifth verification result.

[0055] For example, in an insurance scenario, the associated data of "PDF application form + Excel premium table" is compared with the corresponding business data in "PDF underwriting conclusion + Excel endorsement table" to check whether the association logic of "insured amount and underwriting limit" and "premium calculation and endorsement adjustment" is consistent; in a medical scenario, the associated data of "PDF medical record + Excel test report" is compared with the corresponding data in "PDF discharge summary + Excel medication list" to verify whether the association relationship between "diagnosis and medication" is reasonable, thus obtaining the fifth verification result.

[0056] S354. Analyze the fourth and fifth verification results. If there is a mismatch in the business data association logic, or the deviation of the association field combination result exceeds the preset threshold range, then mark the corresponding first data field as the second data field.

[0057] Specifically, when there are contradictions in the association logic of business data (for example, the matching relationship between "insured amount" and "underwriting limit" is incorrect in insurance, or "diagnosis conclusion" and "medication list" are not related in medical care), or the deviation of the combined results of associated fields exceeds the threshold (for example, the difference between the premium calculation result and the amount after the endorsement adjustment exceeds the limit), the corresponding first data field is marked as the second data field to achieve accurate positioning of the differences in mixed format data.

[0058] As another optional implementation, step S3 may further include steps S355 to S358.

[0059] S355. Based on preset field mapping rules, perform field-level association mapping between fields in the database table and row and column data in the spreadsheet file.

[0060] For example, in an insurance scenario, the "pol_id" and "premium" fields in the database are associated with the "policy number" and "premium amount" columns in the Excel table according to the mapping rules; in a medical scenario, the "patient_id" and "diagnosis_code" fields in the database are mapped with the "patient number" and "diagnosis code" columns in the Excel table to ensure that the fields of different structured carriers can correspond according to the rules.

[0061] S356. Verify each field after the association mapping to obtain the sixth verification result. The verification includes automatic data type conversion and adaptation, and numerical deviation verification based on the preset fault tolerance threshold.

[0062] Among them, automatic data type conversion and adaptation can handle compatibility issues such as "date type" in the database and "text type date" in Excel; numerical deviation verification based on preset fault tolerance threshold can handle minor differences caused by rounding in fields such as "premium" in the database and "actual premium received" in Excel in insurance scenarios (considered consistent within the threshold), and measurement errors in "blood pressure value" in the database and "blood pressure record" in Excel in medical scenarios (qualified within the threshold), ultimately forming the sixth verification result.

[0063] S357. Capture dynamic business data from system interfaces or logs in real time, and automatically match and verify it with preset benchmark data in offline spreadsheet templates to obtain the seventh verification result.

[0064] For example, in the insurance scenario, the "real-time underwriting data" from the underwriting system interface is captured in real time and compared with the "benchmark underwriting parameters" in the offline Excel underwriting template to check whether they are updated synchronously; in the medical scenario, the "real-time test results" from the LIS system log are captured and matched with the "standard reference values" in the offline Excel test template to monitor whether the dynamic data is synchronized with the benchmark, thereby obtaining the seventh verification result.

[0065] S358. Analyze the sixth and seventh verification results. If there is a mismatch in field mapping, a numerical deviation exceeding the fault tolerance threshold, or a discrepancy between dynamic data and baseline data, then mark the corresponding first data field as the second data field.

[0066] Specifically, when there is a mismatch in field mapping (e.g., the "policy status" in the insurance database corresponds incorrectly to the "status code" in Excel), the numerical deviation exceeds the fault tolerance threshold (e.g., the difference between the "blood glucose value" in the medical database and the record in Excel exceeds the limit), or the dynamic data is not synchronized with the baseline data (e.g., the real-time insurance underwriting data is not updated to the offline template according to the rules), the corresponding first data field is marked as the second data field, thereby achieving accurate identification of differences in structured data.

[0067] Through steps S31 to S34 and steps S351 to S358, a multi-layered, full-scenario consistency verification system was constructed, achieving comprehensive verification of multi-source heterogeneous data. For the characteristics of portable documents both within and across documents, it solved the verification challenges caused by key name variations and information fragmentation, ensuring consistency of semantically identical data across documents. For mixed-format data sources, business logic association and cross-source composite verification eliminated data association deviations caused by different format combinations. For structured data, field mapping and dynamic verification solved problems such as data type differences and synchronization delays. Overall, by using preset thresholds and logical rules, inconsistent second data fields were accurately marked, covering diverse scenarios such as single-format, cross-format, and mixed-format data, and achieving deep verification from the field level to the business association level. This provided precise targets for subsequent root cause tracing, significantly improving the reliability and consistency of multi-source heterogeneous data.

[0068] S4. Based on the preset difference analysis rules, perform root cause tracing on the second data field, and determine the difference type corresponding to the second data field based on the root cause tracing results.

[0069] As an alternative implementation method, please continue reading. Figure 4 , Figure 4 This is a schematic diagram illustrating the execution flow of root cause tracing in the multi-source heterogeneous data verification and tracing method provided in this application embodiment, as shown below. Figure 4 As shown, step S4 above can specifically include steps S41 to S45.

[0070] S41. Based on the preset target business data verification rules, the root cause location of the second data field is performed in a preset dimension through the rule engine and semantic reasoning.

[0071] For example, in the insurance scenario, the rule engine loads preset rules such as "premium calculation rules" and "policyholder information verification logic", and analyzes whether the second data field (e.g., "inconsistent premium amount") is caused by rule application deviation through semantic reasoning; in the medical scenario, based on the "diagnosis and medication matching rules", it infers whether "abnormal test results" are due to conflict in diagnosis and treatment logic, thereby locking in the root cause of the difference from the perspective of business rules.

[0072] S42. Obtain multi-version business files and database historical snapshots associated with the second data field through the data lineage analysis mechanism, and construct a data flow link diagram based on the multi-version business files and database historical snapshots.

[0073] Technically, by recording the generation, transmission, and transformation nodes of data (for example, the flow of insurance data from insurance entry to underwriting approval, and the transmission path of medical data from testing to diagnosis), multiple versions of files and historical snapshots are associated with each other in a timeline to form a visual link diagram, which clearly shows the flow nodes and relationships of the second data field throughout its entire life cycle.

[0074] S43. Compare the second data field with the corresponding field in the historical version data, and locate the difference node in the data flow link diagram based on the comparison result.

[0075] For example, in the insurance scenario, by comparing the historical versions of the "premium amount" in the application form, underwriting form, and policy, if the difference first appears in the underwriting process, the underwriting node is marked as the point of difference in the link diagram; in the medical scenario, by tracing the version changes of the "blood glucose value" in test records, doctor's diagnosis, and medical record archiving, the operation node where the first deviation occurs is located, providing a precise range for root cause analysis.

[0076] S44. Based on the data operation logs, data processing scripts, and data permission records of the nodes with differences, determine the cause type of the differences through a preset set of causal association analysis rules.

[0077] The rule set includes judgment logic such as "operation log shows manual modification without approval record → manual entry error", "script execution log has abnormal error report → processing logic defect", and "operation account exceeds the scope of permission → unauthorized modification". For example, in the insurance scenario, if the script log of the "insurance amount difference" node shows an error in the calculation formula, it is judged as a script logic defect; in the medical scenario, if the operator corresponding to the "diagnosis result difference" does not have modification permission, it is judged as an unauthorized operation.

[0078] S45. Generate a traceability analysis conclusion that includes a root cause description, a responsible node identifier, and improvement suggestions based on the type of difference cause, and associate the traceability analysis conclusion with the corresponding difference generation node.

[0079] For example, regarding "manual data entry errors," the conclusion clearly states the error content, the operator, and the suggestion to "add secondary verification"; regarding "script logic defects," it explains the location of the defect, the responsible department, and the solution to "optimize the script algorithm." By linking this to the chain diagram, the causes of differences at each node and the direction for improvement can be visually displayed, providing clear guidance for problem rectification.

[0080] Through steps S41 to S45 above, a complete root cause tracing system for discrepancies was constructed. A rule engine and semantic reasoning were used to pinpoint the root causes of anomalies in the business dimension. Data lineage analysis and link diagrams were used to present the data flow trajectory. Historical comparisons were combined to accurately locate discrepancy nodes. Causal rule sets were used to automatically determine the cause type, ultimately generating a structured conclusion of the association link diagram. This process achieves a closed loop from discrepancy discovery to root cause location, meeting regulatory retrospective requirements and providing precise evidence for data quality optimization, thus improving the controllability of multi-source heterogeneous data management.

[0081] S5. Generate multi-source heterogeneous data verification and tracing results based on business data source, second data field, root cause tracing results and difference type.

[0082] As an optional implementation, step S5 may further include steps S51 to S53.

[0083] S51. According to the preset structured report template, generate initial verification and tracing results based on the source identifier of the business data source, the field content and location information of the second data field, the difference generation nodes and flow link segments in the root cause tracing results, and the classification statistics of the difference types.

[0084] By using a fixed template framework, information such as the source identifier of the business data source (e.g., the number of the insurance PDF application form, the name of the medical database), the specific content and location of the second data field (e.g., "premium amount" in cell A10 of an Excel sheet, "blood glucose value" on page 3 of a PDF medical record), the node information for root cause tracing (e.g., differences appear in the underwriting process, the test report entry process), and the statistics of difference types (e.g., the percentage of manual entry errors, the number of script logic defects) are organized in an orderly manner according to business dimensions, forming a clear initial result, laying the foundation for subsequent visualization and interactive display.

[0085] S52. Visualize the initial verification and traceability results. Visualization includes displaying the inconsistent distribution density of multi-source data using a difference heatmap and highlighting the links where differences occur.

[0086] Among them, the difference heatmap intuitively displays the distribution density of inconsistent fields in multi-source data through color intensity (for example, densely differentiated areas of the "beneficiary information" field in insurance data, and high-frequency difference points of the "medication record" field in medical data); the link node highlighting uses eye-catching colors to mark the links where differences occur in the data flow diagram (for example, underwriting system nodes, laboratory department data entry nodes), allowing users to quickly identify areas with concentrated problems and key links, and improve the efficiency of perception of overall data quality.

[0087] S53. Add an interactively displayed data flow link diagram and compare historical data versions to the initial verification and traceability results to obtain multi-source heterogeneous data verification and traceability results.

[0088] The interactive link diagram allows users to click on nodes to view detailed differences, operation logs, and other related information for that stage. Historical version comparison records can be viewed through timeline scrolling and version switching, visually displaying the changes in the second data field at different points in time (e.g., records of insurance premium amount modifications or version iterations of medical diagnosis results). The resulting verification and traceability results retain the rigor of structured reports while satisfying the query needs of different users for detailed information through interactive functions, thus improving the practicality and operability of the results.

[0089] The multi-source heterogeneous data verification and traceability method provided in this application achieves efficient processing and accurate traceability of multi-source heterogeneous data by constructing a full-link technology system from data acquisition to result generation. It breaks down data barriers through targeted format parsing and semantic mapping, accurately identifies inconsistent fields using a layered verification mechanism, and achieves root cause localization by combining a rule engine, data lineage analysis, and causal association rules. Finally, it presents the results in a structured and visual report. This method improves the efficiency and accuracy of multi-source data verification, strengthens the traceability of data throughout its entire lifecycle, and provides strong support for compliance management and data quality control in fields such as finance and healthcare.

[0090] Please continue reading. Figure 5 , Figure 5 This is a schematic diagram of the system structure of the multi-source heterogeneous data verification and traceability device provided in the embodiments of this application, as shown below. Figure 5 As shown, the multi-source heterogeneous data verification and traceability device 50 includes: a multi-source heterogeneous data acquisition module 51, a multi-source heterogeneous data preprocessing module 52, a data consistency verification module 53, a data root cause tracing module 54, and a verification and traceability result module 55.

[0091] The multi-source heterogeneous data acquisition module 51 is specifically used to acquire multi-source heterogeneous data in a preset data format from the business data source of the preset target business. The multi-source heterogeneous data contains at least two different data formats of business data.

[0092] The multi-source heterogeneous data preprocessing module 52 is specifically used to preprocess the multi-source heterogeneous data, extract the first data field from each data source, and match the first data field with the same semantic meaning in different data sources based on preset semantic mapping rules.

[0093] The data consistency verification module 53 is specifically used to perform consistency verification on the first data field after matching, and obtain a second data field whose verification result is inconsistent.

[0094] The data root cause tracing module 54 is specifically used to perform root cause tracing on the second data field based on preset difference analysis rules, and determine the difference type corresponding to the second data field based on the root cause tracing result.

[0095] The verification and traceability result module 55 is specifically used to generate multi-source heterogeneous data verification and traceability results based on the business data source, the second data field, the root cause tracing result, and the difference type.

[0096] As an optional implementation, the multi-source heterogeneous data preprocessing module 52 is further specifically used to extract business information from the multi-source heterogeneous data in the portable document format using optical character recognition to obtain the first data field scattered across different pages; to perform table structure parsing on the multi-source heterogeneous data in the spreadsheet format using a preset table data parsing tool to obtain the first data field in the cell; and to perform field mapping transformation on the multi-source heterogeneous data in the database table format based on a preset mapping relationship table between database fields and business fields, and to perform data type standardization long distance to obtain the standardized first data field.

[0097] As an optional implementation, the data consistency verification module 53 is further specifically used to compare the same semantic key value matched and confirmed in different portable documents field by field, verify whether the key value corresponding to the key name expression variant is consistent, and obtain a first verification result; if the portable document contains cross-page distribution information, associate and splice the key value information of the same business object scattered on different pages, and then perform consistency verification on the associated key value information to obtain a second verification result; compare the first data field extracted from the portable document with the same semantic field matched in the non-portable document to obtain a third verification result; analyze the first verification result, the second verification result and the third verification result, and if there is a key value mismatch or the key value deviation exceeds a preset threshold range, then mark the corresponding first data field as the second data field.

[0098] As an optional implementation, the data consistency verification module 53 is further specifically used to extract the first data field from a mixed format data source composed of portable document format and tabular format, and generate a corresponding business dataset according to the business logic of the preset target business; through an automated verification script, the same semantic data field in the business dataset and the target portable document is compared item by item to verify the correspondence and numerical consistency between the associated data items and the target declared data items, and a fourth verification result is obtained; the associated data in the mixed format data source is subjected to cross-source composite verification with the corresponding business data in the mixed format data source composed of portable document format and tabular format, and a fifth verification result is obtained; the fourth verification result and the fifth verification result are analyzed, and if there is a mismatch in the business data association logic, or the deviation of the associated field combination result exceeds a preset threshold range, the corresponding first data field is marked as the second data field.

[0099] As an optional implementation, the data consistency verification module 53 is further specifically used to perform field-level association mapping between fields in the database table and row and column data in the spreadsheet file based on preset field mapping rules; to perform item-by-item verification on the mapped fields to obtain a sixth verification result, wherein the item-by-item verification includes automatic data type conversion and adaptation, and numerical deviation verification based on a preset fault tolerance threshold; to capture dynamic business data in the system interface or logs in real time, and to perform automatic matching and verification with preset benchmark data in the offline spreadsheet template to obtain a seventh verification result; to analyze the sixth verification result and the seventh verification result, and if there is a field mapping mismatch, a numerical deviation exceeding the fault tolerance threshold, or dynamic data being out of sync with the benchmark data, then the corresponding first data field is marked as the second data field.

[0100] As an optional implementation, the data root cause tracing module 54 is further specifically used to locate the root cause of the second data field in a preset dimension based on the preset target business data verification rules, through a rule engine and semantic reasoning; to obtain multi-version business files and database historical snapshots associated with the second data field through a data lineage analysis mechanism, and to construct a data flow link diagram based on the multi-version business files and database historical snapshots; to perform a traceability comparison between the second data field and the corresponding field in the historical version data, and to locate the difference generation node in the data flow link diagram based on the traceability comparison result; to determine the difference cause type based on the data operation logs, data processing scripts, and data permission records of the difference generation node through a preset causal association analysis rule set; to generate a traceability analysis conclusion containing a root cause description, a responsible node identifier, and improvement suggestions based on the difference cause type, and to associate the traceability analysis conclusion with the corresponding difference generation node.

[0101] As an optional implementation, the verification and traceability result module 55 is further specifically used to generate initial verification and traceability results according to a preset structured report template, based on the source identifier of the business data source, the field content and location information of the second data field, the difference-generating nodes and flow link segments in the root cause tracing results, and the classification statistics of the difference types; to perform visualization processing on the initial verification and traceability results, the visualization processing including a difference heatmap to display the inconsistency distribution density of multi-source data, highlighting the difference-generating links in the link nodes; and to add an interactively displayable data flow link diagram and a comparison record of historical data versions to the initial verification and traceability results to obtain the multi-source heterogeneous data verification and traceability results.

[0102] It should be noted that the above-mentioned multi-source heterogeneous data verification and traceability device can execute the multi-source heterogeneous data verification and traceability method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in the embodiments of the multi-source heterogeneous data verification and traceability device can be found in the multi-source heterogeneous data verification and traceability method provided in the embodiments of this application.

[0103] Figure 6 This is a schematic diagram of the hardware structure of the electronic device for executing the multi-source heterogeneous data verification and traceability method provided in the embodiments of this application, as shown below. Figure 6 As shown, the electronic device 600 includes:

[0104] One or more processors 610 and memory 620, Figure 6 Take the 610 processor as an example.

[0105] The processor 610 and the memory 620 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0106] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the multi-source heterogeneous data verification and traceability method in the embodiments of this application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 620, thereby realizing the multi-source heterogeneous data verification and traceability method in the above-described method embodiments.

[0107] The memory 620 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the multi-source heterogeneous data verification and traceability device. Furthermore, the memory 620 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include memory remotely located relative to the processor 610, and these remote memories can be connected to the multi-source heterogeneous data verification and traceability device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0108] The one or more modules are stored in the memory 620. When executed by the one or more processors 610, they perform the multi-source heterogeneous data verification and traceability method in any of the above method embodiments. For example, they perform the methods described above. Figure 2 Method steps S1 to S5, Figure 3 Method steps S31 to S34, Figure 4 Steps S41 to S45 in the method are implemented. Figure 5 The functions of modules 51-55 in the document.

[0109] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0110] This application provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 6 One of the processors 610 can enable the above-described one or more processors to execute the multi-source heterogeneous data verification and traceability method in any of the above method embodiments, for example, to execute the method described above. Figure 2 Method steps S1 to S5, Figure 3 Method steps S31 to S34, Figure 4 Steps S41 to S45 in the method are implemented. Figure 5 The functions of modules 51-55 in the document.

[0111] This application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by an electronic device, enable the electronic device to perform the multi-source heterogeneous data verification and traceability method described in any of the above method embodiments. For example, it executes the method described above. Figure 2 Method steps S1 to S5, Figure 3 Method steps S31 to S34, Figure 4 Steps S41 to S45 in the method are implemented. Figure 5 The functions of modules 51-55 in the document.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for verifying and tracing multi-source heterogeneous data, characterized in that, include: Obtain multi-source heterogeneous data in a preset data format from the business data source of the preset target business, wherein the multi-source heterogeneous data contains at least two different data formats of business data; The multi-source heterogeneous data is preprocessed, and the first data field in each data source is extracted. Based on the preset semantic mapping rules, the first data field with the same semantic meaning in different data sources is matched. Perform a consistency check on the first data field after matching to obtain a second data field that is inconsistent with the check result; The second data field is traced back to its root cause based on the preset difference analysis rules, and the difference type corresponding to the second data field is determined based on the root cause tracing results. Multi-source heterogeneous data verification and tracing results are generated based on the business data source, the second data field, the root cause tracing result, and the difference type.

2. The method for verifying and tracing multi-source heterogeneous data according to claim 1, characterized in that, The preset data format includes portable document format, spreadsheet format, and database table format. The step of preprocessing the multi-source heterogeneous data and extracting the first data field from each data source includes: Business information is extracted from the multi-source heterogeneous data in the portable document format using optical character recognition to obtain the first data field scattered across different pages; The multi-source heterogeneous data in the spreadsheet format is parsed using a preset table data parsing tool to obtain the first data field in the cell; Based on a preset mapping table between database fields and business fields, the multi-source heterogeneous data in the database table format is transformed by field mapping and data type standardization long distance to obtain the standardized first data field.

3. The method for verifying and tracing multi-source heterogeneous data according to claim 1, characterized in that, The step of performing a consistency check on the matched first data field to obtain a second data field whose check result is inconsistent includes: For the same semantic key value that has been matched and confirmed in different portable documents, each field is compared to verify whether the key value corresponding to the key name expression variant is consistent, and the first verification result is obtained. If the portable document contains cross-page distribution information, the key-value information of the same business object scattered across different pages is associated and spliced ​​together, and then the consistency of the associated key-value information is checked to obtain a second check result; The first data field extracted from the portable document is compared with the same semantic field matched in the non-portable document using a cross-format comparison to obtain a third verification result. Analyze the first verification result, the second verification result, and the third verification result. If there is a key-value mismatch or the key-value deviation exceeds a preset threshold range, then mark the corresponding first data field as the second data field.

4. The method for verifying and tracing multi-source heterogeneous data according to claim 1, characterized in that, The step of performing a consistency check on the matched first data field to obtain a second data field whose check result is inconsistent includes: The first data field is extracted from a mixed format data source consisting of portable document format and tabular format, and a corresponding business dataset is generated according to the business logic of the preset target business. The automated verification script compares the same semantic data field in the business dataset and the target portable document item by item to verify the correspondence between the associated data items and the target declared data items and the consistency of the values, thus obtaining the fourth verification result. The associated data in the hybrid format data source is subjected to cross-source composite verification with the corresponding business data in the hybrid format data source composed of portable document format and tabular format to obtain the fifth verification result. Analyzing the fourth and fifth verification results, if there is a mismatch in the business data association logic, or if the deviation of the association field combination result exceeds a preset threshold range, then the corresponding first data field is marked as the second data field.

5. The method for verifying and tracing multi-source heterogeneous data according to claim 1, characterized in that, The step of performing a consistency check on the matched first data field to obtain a second data field whose check result is inconsistent includes: Based on preset field mapping rules, the fields in the database table are mapped to the rows and columns in the spreadsheet file at the field level. The fields after the association mapping are verified item by item to obtain the sixth verification result. The item by item verification includes automatic data type conversion and adaptation, and numerical deviation verification based on a preset fault tolerance threshold. Real-time capture of dynamic business data from system interfaces or logs, automatic matching and verification with preset benchmark data in offline spreadsheet templates, and obtaining the seventh verification result; Analyze the sixth and seventh verification results. If there is a mismatch in field mapping, a numerical deviation exceeding the fault tolerance threshold, or a discrepancy between dynamic data and baseline data, then mark the corresponding first data field as the second data field.

6. The method for verifying and tracing multi-source heterogeneous data according to claim 1, characterized in that, The step of performing root cause analysis on the second data field based on preset difference analysis rules includes: Based on the data verification rules of the preset target business, the root cause location of the second data field is performed in preset dimensions through the rule engine and semantic reasoning. The data lineage analysis mechanism is used to obtain multiple versions of business files and database historical snapshots associated with the second data field, and a data flow link diagram is constructed based on the multiple versions of business files and database historical snapshots; The second data field is compared with the corresponding field in the historical version data, and the difference node is located in the data flow link diagram based on the comparison result. Based on the data operation logs, data processing scripts, and data permission records of the nodes that caused the differences, the cause type of the differences is determined through a preset set of causal association analysis rules. Based on the type of cause of the difference, a traceability analysis conclusion is generated that includes a root cause description, a responsible node identifier, and improvement suggestions. The traceability analysis conclusion is then associated with the corresponding node that generated the difference.

7. The method for verifying and tracing multi-source heterogeneous data according to claim 6, characterized in that, The step of generating multi-source heterogeneous data verification and tracing results based on the business data source, the second data field, the root cause tracing result, and the difference type includes: According to the preset structured report template, the initial verification and tracing results are generated based on the source identifier of the business data source, the field content and location information of the second data field, the difference generation nodes and flow link segments in the root cause tracing results, and the classification statistics of the difference types. The initial verification and traceability results are visualized, including a difference heatmap to show the inconsistency distribution density of multi-source data and highlighting the links to mark the points where differences occur. The interactively displayed data flow diagram and the comparison record of historical data versions are added to the initial verification and traceability results to obtain the multi-source heterogeneous data verification and traceability results.

8. A multi-source heterogeneous data verification and traceability device, characterized in that, include: A multi-source heterogeneous data acquisition module is used to acquire multi-source heterogeneous data in a preset data format from a business data source of a preset target business. The multi-source heterogeneous data contains at least two different data formats of business data. A multi-source heterogeneous data preprocessing module is used to preprocess the multi-source heterogeneous data, extract the first data field from each data source, and match the first data field with the same semantic meaning in different data sources based on preset semantic mapping rules. The data consistency verification module is used to perform consistency verification on the first data field after matching, and obtain the second data field that is inconsistent in the verification result. The data root cause tracing module is used to perform root cause tracing on the second data field based on preset difference analysis rules, and determine the difference type corresponding to the second data field based on the root cause tracing results. The verification and traceability result module is used to generate multi-source heterogeneous data verification and traceability results based on the business data source, the second data field, the root cause tracing result, and the difference type.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the multi-source heterogeneous data verification and traceability method according to any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by an electronic device, cause the electronic device to perform the multi-source heterogeneous data verification and traceability method according to any one of claims 1-7.

Images