A system and method for real-time traceability auditing and supervision of multi-source data

Abstract

The application discloses a kind of for multi-source data real-time tracing supervision system and method, it is related to data management technical field, the method includes: when data is generated, the original characteristic information of data is automatically collected, write into blockchain and generate unique data blood relationship identifier, establish data blood relationship index table;When data is transferred, real-time record node identification, time stamp and operation type, obtain attenuation value from attenuation coefficient control table, calculate current confidence coefficient, as new node is chained and forms chain connection;Periodic check data integrity, when detecting data quality anomaly, call the complete blood relationship chain corresponding to abnormal data, associate each link attenuation value and offset cumulative amount;Filter the link of confidence degree drop amplitude greater than threshold value, combined with characteristic offset jump point, call the chaining record of maximum attenuation point, locate specific operation behavior.The application accurately locates the maximum attenuation point and specific operation behavior of data anomaly, improves the efficiency of supervision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data governance, and particularly relates to a multi-source data real-time traceability auditing and supervision system and method. BACKGROUND

[0002] With the rapid development of information technology, data presents the characteristics of multi-sourcing, heterogeneity and frequent flow, and data is prone to distortion, tampering or quality abnormalities in the links of generation, transmission and processing, which brings challenges to data credibility and security. The existing data traceability method relies on a centralized system, which has a single point of failure risk and is difficult to achieve full-link trusted recording. The traditional auditing method is mostly post-event static verification, which lacks real-time and dynamic adjustment capabilities, and cannot accurately locate the root cause of data abnormalities. At the same time, the existing technology lacks quantitative measurement of the credibility in the data flow process, and it is difficult to evaluate the degree of data quality degradation through a unified standard, resulting in low data supervision efficiency, delayed abnormality traceability, and inability to meet the demand for real-time supervision of multi-source data. SUMMARY

[0003] The present application relates to the technical field of data governance, and particularly relates to a multi-source data real-time traceability auditing and supervision system and method.

[0004] To achieve the above-mentioned purpose, the present application provides the following technical scheme: a multi-source data real-time traceability auditing and supervision method, which comprises:

[0005] Deploying a blockchain node at the data source, when data is generated, automatically collecting the original feature information of the data, writing it into the blockchain and generating a unique data blood relationship identifier, and establishing a data blood relationship index table;

[0006] When the data flows, the node identifier, timestamp and operation type are recorded in real time, the attenuation value is obtained from the attenuation coefficient table according to the operation type and complexity, the current credibility coefficient is calculated, and the flow node information, attenuation value and current credibility coefficient are used as a new node to form a chain association;

[0007] Periodically checking the data integrity, when detecting data quality abnormalities, extracting the current feature information of the abnormal data, calling the complete blood relationship chain corresponding to the abnormal data, analyzing the field mapping deviation degree and value range deviation degree, and associating the attenuation value and offset cumulative amount of each link;

[0008] Filtering the link with a credibility drop greater than a threshold value, marking it as a high-risk attenuation point, determining the maximum attenuation point in combination with the feature offset jump point, calling the on-chain record of the maximum attenuation point, and locating the specific operation behavior;

[0009] Visualizing the root cause link, data blood relationship chain and feature offset change curve, using the traceability conclusion as an auditing record on the chain, and dynamically updating the attenuation coefficient table.

[0010] According to the above scheme, the original feature information includes a data field structure, a field value range, a generation timestamp, an acquisition device identifier, and an initial trust coefficient;

[0011] The original feature information is serialized according to a predefined structured format, a feature digest is generated by performing a hash operation on the serialized original feature information, and the data blood relationship identifier is combined with the current blockchain state information to generate a data blood relationship identifier, and the complete feature information is written into a new block of the blockchain;

[0012] The data blood relationship index table includes a data blood relationship identifier field, an original data storage address field, and a blockchain transaction hash field; a mapping relationship between the data blood relationship identifier and the original data physical storage address is established, and an index relationship between the data blood relationship identifier and the blockchain transaction record is established.

[0013] According to the above scheme, the initial trust coefficient includes:

[0014] The initial trust coefficient is obtained through a multi-dimensional feature evaluation model, and is comprehensively evaluated based on a data type factor, a data source authentication level, a data integrity check value, a timestamp trustworthiness, and a data generation frequency, and the formula is as follows:

[0015] C0=w1×DT+w2×DA+w3×DC+w4×TR+w5×DF;

[0016] Wherein, C0 represents the initial trust coefficient; w1, w2, w3, w4 and w5 represent the weighted coefficients; DT represents the data type factor; DA represents the data source authentication level; DC represents the data integrity check value; TR represents the timestamp trustworthiness; DF represents the data generation frequency;

[0017] The timestamp trustworthiness TR is obtained according to the time difference between the data generation time and the blockchain writing time, and the formula is TR=e (-λ×Δt) Wherein, λ represents the attenuation coefficient;

[0018] The data type factor is divided according to the normativity of the data structure; the data source authentication level is determined by the validity of the digital certificate of the data source device and the access control level; the data integrity check value is calculated based on the SHA-3, BLAKE3 or SHA-256 algorithm to match the checksum; the data generation frequency is the number of data generated per unit time, which is used to evaluate the stability of data generation.

[0019] According to the above scheme, the data flow includes data through the collection, transmission, storage or sharing link;

[0020] The complexity is evaluated based on the number of fields involved in the operation, the depth of calculation logic and the number of associated data records, and is divided into three levels of simple, medium and complex; simple operations include data format conversion, field renaming and data filtering; medium operations include data aggregation, association query and basic statistical analysis; complex operations include machine learning model training, advanced data mining, multi-dimensional data analysis and complex algorithm application;

[0021] The attenuation value is obtained from the attenuation coefficient table according to the operation type and complexity, and the current credibility coefficient is calculated, and the formula is as follows:

[0022] C t = C t-1 × (1-α);

[0023] Wherein, C t represents the credibility coefficient of the current node; C t-1 represents the credibility coefficient of the last node, and α represents the attenuation value corresponding to the current operation;

[0024] The attenuation coefficient table is obtained by analyzing historical data governance cases, and the average data distortion rate under different operation types is counted, and the reference attenuation coefficient is set according to the international standard;

[0025] The flow node information, attenuation value and current credibility coefficient are serialized according to the pre-defined structured format, the feature digest is generated by hash operation on the serialized information, and the new data flow node identifier is generated by combining the current blockchain state information, and the complete flow node information, attenuation value, current credibility coefficient, feature digest and data flow node identifier are written into the new block of the blockchain, forming a chain association with the last node;

[0026] The flow node information includes the physical address, logical address, device identifier, system identifier, user identifier, operation IP address, operation port number, operation protocol type, operation command, operation parameter, operation result status code and operation result description of the operation node.

[0027] According to the above scheme, the chain association adopts the Merkle tree structure, each data flow node contains a pointer to its parent node, and the complete data flow path can be obtained by tracing back level by level; at the same time, a data flow index table is established in the blockchain, the data flow index table includes a data blood relationship identifier field, a data flow node identifier field, a flow time stamp field, an operation type field, an attenuation value field and a credibility coefficient field, a mapping relationship between the data blood relationship identifier and the data flow node is established, and a time sequence association relationship between the data flow nodes is established.

[0028] According to the above scheme, the data integrity includes data field structure, field value range, data relationship constraint and data integrity constraint;

[0029] When the data quality anomaly is detected, an abnormal processing procedure is automatically triggered to extract current characteristic information of the abnormal data, the current characteristic information including data field structure, field value range, current timestamp, current storage node identifier, and current trustworthiness coefficient;

[0030] The current characteristic information of the abnormal data is compared with the original characteristic information to analyze field mapping deviation degree and value range deviation degree; the field mapping deviation degree is obtained by calculating deviation degree of the current field structure and the original field structure, and the deviation degree includes field quantity difference, field name difference, field type difference, field sequence difference, and field constraint condition difference; the value range deviation degree is obtained by calculating deviation degree of the current field value range and the original field value range, and the deviation degree includes value domain upper limit offset, value domain lower limit offset, value domain distribution form offset, and value domain dispersion degree offset;

[0031] The field mapping deviation degree and the value range deviation degree are associated with the attenuation values of each flow transfer link for correlation analysis to calculate offset accumulations of each link, the offset accumulations including field mapping deviation accumulation, value range deviation accumulation, and trustworthiness attenuation accumulation.

[0032] According to the above scheme, a time sequence analysis algorithm is used to perform time sequence correlation analysis on the attenuation values, characteristic offset amounts, and offset accumulations of each link to identify change trend, mutation point, and periodic fluctuation law of the characteristic offset amount; by comparing the trustworthiness drop amplitude of each link with a preset threshold value, a link with a trustworthiness drop amplitude greater than the threshold value is screened out and marked as a high-risk attenuation point;

[0033] The trustworthiness drop amplitude threshold value is obtained by collecting trustworthiness coefficient change records of each link in the running process, distinguishing trustworthiness natural attenuation range of normal data flow transfer, and determining trustworthiness drop amplitude interval of abnormal data; a statistical analysis method is used to determine a critical value of normal attenuation and abnormal attenuation, and the critical value is used as an initial trustworthiness drop amplitude threshold value;

[0034] In combination with the position and amplitude of the characteristic offset jump point, an attenuation point with the greatest impact on data quality is determined, and the characteristic offset jump point is obtained by calculating change rate of characteristic offset amount of adjacent links, and a point with a change rate greater than a preset jump threshold value is marked as a characteristic offset jump point;

[0035] The jump threshold value is obtained by extracting characteristic offset amount change rates of each link in historical data, statistically analyzing change rate distribution range under normal operation, and determining change rate peak value corresponding to abnormal operation; the jump threshold value is determined by ROC curve analysis;

[0036] Call the uplink record of the maximum attenuation point, the uplink record including the physical address, logical address, device identification, system identification, user identification, operation IP address, operation port number, operation protocol type, operation command, operation parameter, operation result status code and operation result description of the operation node; through analyzing the uplink record, the specific operation behavior causing the data quality anomaly is located.

[0037] According to the above scheme, the field mapping deviation degree is as follows:

[0038] D m =(n1+n2+n3+n4+n5) / N;

[0039] Wherein, D m is the field mapping deviation degree; n1 is the field quantity difference value, i.e. the absolute value of the difference between the current field quantity and the original field quantity; n2 is the field name difference value, i.e. the number of name-mismatched fields; n3 is the field type difference value, i.e. the number of type-mismatched fields; n4 is the field order difference value, i.e. the number of fields with inconsistent order with the original; n5 is the field constraint condition difference value, i.e. the number of constraint condition-mismatched fields; and N is the total number of original fields.

[0040] The value range deviation degree is as follows:

[0041] D r =(|U c -U o |+|L c -L o |+|F c -F o |+|S c -S o |) / (U o +L o +F o +S o );

[0042] Wherein, D r is the value range deviation degree; U c is the upper limit of the current field value domain; U o is the upper limit of the original field value domain; L c is the lower limit of the current field value domain; L o is the lower limit of the original field value domain; F c is the current field value domain distribution form parameter; F o is the original field value domain distribution form parameter; S c is the current field value domain dispersion degree parameter; and S o is the original field value domain dispersion degree parameter.

[0043] Characteristic offset change rate, formula as follows:

[0044] R = (F t -F t-1 ) / F t-1 ;

[0045] Wherein, R represents the characteristic offset change rate; F t represents the characteristic offset of the current link; F t-1 represents the characteristic offset of the previous link;

[0046] Offset cumulative amount, formula as follows:

[0047] C m =ΣD mi ; C r =ΣD ri ; C a =Σα i ;

[0048] Wherein, C m represents the field mapping deviation cumulative amount; D mi represents the field mapping deviation degree of the ith link; C r represents the value range deviation cumulative amount; D ri represents the value range deviation degree of the ith link; C a represents the credibility attenuation cumulative amount; alpha i represents the attenuation value of the ith link.

[0049] According to the above scheme, the traceability conclusion includes abnormal data identification, node information of root cause link, characteristic offset analysis result, credibility change trend, operation behavior positioning result and rectification suggestion; the traceability conclusion is serialized according to the pre-defined structured format, the serialized traceability conclusion is subjected to hash operation to generate conclusion digest, the conclusion digest is combined with the current blockchain state information to generate audit record identifier, and the complete traceability conclusion, conclusion digest and audit record identifier are written into the new block of the blockchain;

[0050] The dynamic updating of the attenuation coefficient reference table comprises: calculating the actual attenuation influence coefficient of each type of operation based on the operation type, characteristic deviation and credibility change data in the traceability conclusion, comparing the corresponding value in the original attenuation coefficient reference table, and updating the attenuation value of the corresponding operation type and complexity combination when the deviation exceeds the preset adjustment threshold; the adjustment threshold is determined by calculating the deviation distribution of the actual attenuation influence coefficient in the historical traceability conclusion and the value in the original attenuation coefficient reference table; the adjustment threshold is obtained by setting a hierarchical adjustment standard in combination with the weight of the attenuation coefficient in the credibility evaluation; and the updated attenuation coefficient reference table is serialized and subjected to hash operation to generate a reference table abstract, which is written into the blockchain together with the update timestamp and operation node identifier.

[0051] A multi-source data real-time traceability auditing and supervision system, comprising a data acquisition module, a data quality verification module, a credibility evaluation module, a traceability auditing module and a dynamic optimization module.

[0052] The data acquisition module comprises an original data acquisition module and a flow information acquisition module; the original data acquisition module automatically acquires original characteristic information, performs serialization and hash operation to generate a characteristic abstract, and generates a unique data blood relationship identifier; and the flow information acquisition module acquires flow information in real time, generates a new node identifier and chains it.

[0053] The data quality verification module comprises an integrity verification module and an abnormality analysis module; the integrity verification module performs integrity verification on the data according to a preset period, and triggers an abnormality processing procedure when an abnormality is detected; and the abnormality analysis module extracts the current characteristic information of the abnormal data, calls the corresponding complete blood relationship chain, and locates the specific operation behavior causing the abnormality.

[0054] The credibility evaluation module is used for initial evaluation and dynamic calculation of the data credibility coefficient, and quantifies the credibility change in the data flow process.

[0055] The traceability auditing module comprises a visualization module and a record chaining module; the visualization module outputs the root cause link, data blood relationship chain and characteristic deviation change curve through a graphical interface, and intuitively presents the data abnormality traceability result; and the record chaining module serializes the traceability conclusion, generates a hash abstract, combines the blockchain state information to generate an auditing record identifier, and chains it.

[0056] The dynamic optimization module dynamically optimizes the attenuation coefficient reference table based on the traceability result, and improves the accuracy of the credibility evaluation.

[0057] Compared with the prior art, the present application has the following advantages:

[0058] 1. The present application realizes timely discovery and rapid response of data quality abnormalities by recording data flow information in real time and calculating credibility coefficients, in combination with periodic integrity verification.

[0059] 2. This invention quantifies the field mapping deviation, value range deviation, and feature offset change rate, combined with credibility decay analysis, to accurately locate the maximum decay point and specific operation behavior of data anomalies, thereby improving audit efficiency;

[0060] 3. The present invention dynamically updates the attenuation coefficient comparison table based on the traceability conclusion, so that the credibility assessment model is continuously optimized with the actual application scenario, thereby improving the accuracy and adaptability of long-term data supervision. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a structural diagram of a multi-source data real-time traceability audit and supervision system according to the present invention;

[0062] Figure 2 A flowchart of the steps for establishing a data lineage index table for a method for real-time traceability, auditing, and supervising multi-source data according to the present invention;

[0063] Figure 3 This is a flowchart of the steps of data flow analysis for a method for real-time traceability, auditing and supervision of multi-source data according to the present invention;

[0064] Figure 4 The present invention is a flowchart of the steps of locating abnormal data in a method for real-time traceability, auditing and supervision of multi-source data. DETAILED DESCRIPTION

[0065] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0066] Example: Figures 1-4 As shown, the present invention provides a technical solution, a method for real-time traceability auditing and supervision of multi-source data, the method comprising the steps of:

[0067] S1. Deploy blockchain nodes at the source of data. When data is generated, automatically collect the original feature information of the data, write it into the blockchain, generate a unique data lineage identifier, and establish a data lineage index table.

[0068] Specifically, the original feature information includes the data field structure, field value range, generation timestamp, collection device identification and initial credibility coefficient; the original feature information is serialized according to a predefined structured format, and the serialized original feature information is hashed to generate a feature summary, which is combined with the current blockchain status information to generate a data lineage identifier, and the complete feature information is written to the new block of the blockchain; the data lineage index table includes the data lineage identifier field, the original data storage address field and the blockchain transaction hash field; a mapping relationship between the data lineage identifier and the physical storage address of the original data is established, as well as an index relationship between the data lineage identifier and the blockchain transaction record.

[0069] For example, deploying blockchain nodes at the source of data generation to collect and store raw data features. When data is generated, the following information is automatically collected:

[0070] Data field structure: includes field A, field B, field C and field D;

[0071] Field value range: field A∈[1,100], field C∈[0,1000];

[0072] Generation timestamp: 2025-01-01 00:00:00;

[0073] Collection device identification: source server device number;

[0074] Furthermore, the initial credibility coefficient includes:

[0075] The initial credibility coefficient is obtained through a multi-dimensional feature evaluation model, based on a comprehensive evaluation of data type factors, data source authentication level, data integrity check value, timestamp credibility, and data generation frequency. The formula is as follows:

[0076] C0=w1×DT+w2×DA+w3×DC+w4×TR+w5×DF;

[0077] Among them, C0 represents the initial credibility coefficient; w1, w2, w3, w4 and w5 represent weighting coefficients; DT represents the data type factor; DA represents the data source authentication level; DC represents the data integrity check value; TR represents the timestamp credibility; DF represents the data generation frequency;

[0078] The timestamp credibility TR is obtained based on the time difference between the data generation time and the blockchain writing time. The formula is TR=e (-λ×Δt) , where λ represents the attenuation coefficient;

[0079] Data type factor is divided according to the norm of data structure; data source authentication level is determined by the validity of digital certificate of data source device and access control level; data integrity check value is calculated based on SHA-3, BLAKE3 or SHA-256 algorithm to match the checksum; data generation frequency is the number of data generated per unit time, which is used to evaluate the stability of data generation.

[0080] For example: data type factor (DT): structured data, DT = 0.9; data source authentication level (DA): the device holds a second-level digital certificate, and the access control is internal level, DA = 0.8; data integrity check value (DC): SHA-256 is used to calculate the hash value, and the matching degree is 100%, DC = 1.0; timestamp credibility (TR): the time difference Δt between data generation and blockchain writing is 1 second, λ = 0.1, TR = TR = e (-λ×Δt) =0.9048; data generation frequency (DF): 200 pieces are generated per hour, the stability is medium, DF = 0.7; the weighted coefficients w1, w2, w3, w4 and w5 are 0.2, 0.2, 0.3, 0.1, 0.2 respectively, and the calculation is: C0 = 0.2 × 0.9 + 0.2 × 0.8 + 0.3 × 1.0 + 0.1 × 0.9048 + 0.2 × 0.7 = 0.8505, and the initial credibility coefficient is 0.8505.

[0081] S2, when the data flow is transferred, the node identifier, timestamp and operation type are recorded in real time, the attenuation value is obtained from the attenuation coefficient table according to the operation type and complexity, the current credibility coefficient is calculated, and the transferred node information, attenuation value and current credibility coefficient are chained as a new node to form a chain association;

[0082] Specifically, the data flow includes data through the collection, transmission, storage or sharing link;

[0083] The complexity is evaluated based on the number of fields involved in the operation, the depth of calculation logic and the number of associated data records, and is divided into three levels of simple, medium and complex; simple operation includes data format conversion, field renaming and data filtering; medium operation includes data aggregation, association query and basic statistical analysis; complex operation includes machine learning model training, advanced data mining, multi-dimensional data analysis and complex algorithm application; for example: operation type: filtering, simple complexity, involving 2 fields, associated with 50 records;

[0084] According to the operation type and complexity, the attenuation value is obtained from the attenuation coefficient table, and the current credibility coefficient is calculated, and the formula is as follows:

[0085] C t =C t-1 ×(1-α);

[0086] Wherein, Ct Expressed as the credibility coefficient of the current node; C t-1 It is represented as the credibility coefficient of the previous node, and α is represented as the attenuation value corresponding to the current operation;

[0087] The attenuation coefficient comparison table analyzes historical data governance cases, calculates the average data distortion rate under different operation types, and sets the benchmark attenuation coefficient with reference to international standards;

[0088] Serialize the transfer node information, attenuation value, and current credibility coefficient according to a predefined structured format, perform a hash operation on the serialized information to generate a feature summary, and combine it with the current blockchain state information to generate a new data transfer node identifier. Write the complete transfer node information, attenuation value, current credibility coefficient, feature summary, and data transfer node identifier into a new block of the blockchain, forming a chain association with the previous node.

[0089] The transfer node information includes the physical address, logical address, device ID, system ID, user ID, operation IP address, operation port number, operation protocol type, operation command, operation parameters, operation result status code and operation result description of the operation node;

[0090] For example: Attenuation coefficient comparison table: the attenuation value of simple complexity filtering operation α=0.03; credibility coefficient: C t =C0×(1-α)=0.8505×0.97=0.8250; on-chain information: Node ID: processing server number, timestamp: 2025-01-01 00:05:00, operation command: filter records with field A < 10, decay value 0.03, credibility 0.8250, generate a new node and form a chain with the source node;

[0091] In another possible embodiment, when data flows, the node identifier, timestamp and operation type are recorded in real time. The operation type is aggregation, medium complexity, merging the average value of field A and field C; the decay value α=0.06, the corresponding value of the medium complexity aggregation operation; the credibility coefficient is C t =C0×(1-α)=0.8250×(1-0.06)=0.7755; on-chain information: node identification: processing server number, timestamp: 2025-01-01 00:10:00, operation parameters: aggregation period: 5 minutes, forming a chain association. This is only an example and is not a limitation.

[0092] Further, the chain association adopts a Merkel tree structure, each data flow transfer node contains a pointer to its parent node, and the complete data flow transfer path can be obtained by tracing back level by level; meanwhile, a data flow transfer index table is established in the blockchain, the data flow transfer index table includes a data blood relationship identifier field, a data flow transfer node identifier field, a flow transfer timestamp field, an operation type field, a decay value field and a trustworthiness coefficient field, a mapping relationship between the data blood relationship identifier and the data flow transfer node is established, and a time sequence association relationship between the data flow transfer nodes is established.

[0093] S3, periodically checking data integrity, when detecting data quality abnormality, extracting current feature information of abnormal data, calling complete blood relationship chain corresponding to the abnormal data, analyzing field mapping deviation degree and value range deviation degree, and associating decay values and offset accumulations of each link;

[0094] Specifically, the data integrity includes data field structure, field value range, data relationship constraint and data integrity constraint; when detecting data quality abnormality, an abnormal processing procedure is automatically triggered, current feature information of the abnormal data is extracted, the current feature information including data field structure, field value range, current timestamp, current storage node identifier and current trustworthiness coefficient; the current feature information of the abnormal data is compared with original feature information, field mapping deviation degree and value range deviation degree are analyzed; the field mapping deviation degree is obtained by calculating deviation degree of the current field structure and the original field structure, the deviation degree including field number difference, field name difference, field type difference, field order difference and field constraint condition difference; the value range deviation degree is obtained by calculating deviation degree of the current field value range and the original field value range, the deviation degree including value domain upper limit offset, value domain lower limit offset, value domain distribution form offset and value domain dispersion degree offset; the field mapping deviation degree and the value range deviation degree are associated with decay values of each flow transfer link for association analysis, offset accumulations of each link are calculated, the offset accumulations including field mapping deviation accumulation, value range deviation accumulation and trustworthiness decay accumulation.

[0095] Further, the field mapping deviation degree is as follows:

[0096] D m = (n1+n2+n3+n4+n5) / N;

[0097] Wherein, D mThe field mapping deviation degree is represented as n1, the field quantity difference value is represented as n1, that is, the absolute value of the difference between the current field quantity and the original field quantity, n2 represents the field name difference value, that is, the number of fields with name mismatch, n3 represents the field type difference value, that is, the number of fields with type mismatch, n4 represents the field order difference value, that is, the number of fields with inconsistent order, n5 represents the field constraint condition difference value, that is, the number of fields with constraint condition mismatch, and N represents the total number of original fields;

[0098] The value range deviation degree is calculated according to the following formula:

[0099] D r =(|U c -U o |+|L c -L o |+|F c -F o |+|S c -S o |) / (U o +L o +F o +S o );

[0100] Wherein, D r represents the value range deviation degree, U c represents the upper limit of the current field value domain, U o represents the upper limit of the original field value domain, L c represents the lower limit of the current field value domain, L o represents the lower limit of the original field value domain, F c represents the current field value domain distribution form parameter, F o represents the original field value domain distribution form parameter, S c represents the current field value domain dispersion degree parameter, and S o represents the original field value domain dispersion degree parameter.

[0101] For example, periodic verification: integrity verification of data every hour, using BLAKE3 algorithm to compare the hash values of data of each node; abnormality detection: when the verification is performed at 2025-01-01 10:00:00, it is found that the value range of field C of the secondary processing node is abnormal, the current field C ∈ [0, 1500], deviating from the original range [0, 1000], triggering the abnormality processing procedure;

[0102] Extracting current feature information: field value range: field C ∈ [0, 1500]; current timestamp: 2025-01-01 10:00:00; current credibility coefficient: 0.7755; storage node identifier: processing server number;

[0103] For example: call the complete blood chain: through the data blood identifier, trace the full link record of the source; calculate the field mapping deviation D m : the number of original fields N=4, the number of current fields is consistent (n1=0); the field name and type have no difference (n2=n3=0); the field sequence difference: the position of field C and field D is exchanged (n4=1); the field constraint condition difference: the constraint of field C is changed to field C≤1500, and the original is field C≤1000, n5=1; D m =(0+0+0+1+1) / 4=0.5;

[0104] Calculate the value range deviation D r : the upper limit of the value range U o =1000, U c =1500; the lower limit of the value range L o =0 and L c =0 (no deviation); the distribution form parameter F o =2 (normal distribution), F c =2 (no deviation); the dispersion degree parameter S o =200, S c =300; obtain the value range deviation D r =(500+0+0+100) / (1000+0+2+200)=600 / 1202≈0.4992.

[0105] Offset cumulative amount: secondary processing link: C m =0.5, C r =0.4992, C a =0.03+0.06=0.09.

[0106] S4, screen the link with a credibility drop greater than the threshold value, mark it as a high-risk attenuation point, determine the maximum attenuation point in combination with the feature offset jump point, call the upper chain record of the maximum attenuation point, and locate the specific operation behavior;

[0107] Specifically, a time sequence analysis algorithm is used to perform time sequence correlation analysis on the attenuation values, feature offset amounts and offset cumulative amounts of each link, identify the change trend, mutation point and periodic fluctuation law of the feature offset amount; by comparing the credibility drop of each link with the preset threshold value, the link with a credibility drop greater than the threshold value is screened out and marked as a high-risk attenuation point;

[0108] The credibility drop threshold value is obtained by collecting the credibility coefficient change records of each link in the running process, distinguishing the credibility natural attenuation range of normal data flow and the credibility drop interval of abnormal data; a statistical analysis method is used to determine the critical value of normal attenuation and abnormal attenuation, and the critical value is taken as the initial credibility drop threshold value;

[0109] The position and amplitude of the feature offset jump point are combined to determine the decay point that has the greatest impact on data quality. The feature offset jump point is obtained by calculating the change rate of the feature offset of adjacent links. The point whose change rate exceeds the preset jump threshold is marked as the feature offset jump point.

[0110] The jump threshold is determined by extracting the change rate of the feature offset of each link in historical data, counting the change rate distribution range under normal operation, and the change rate peak value corresponding to abnormal operation. The jump threshold is determined by ROC curve analysis.

[0111] The uplink record of the maximum decay point is retrieved, and the uplink record includes the physical address, logical address, device identifier, system identifier, user identifier, operation IP address, operation port number, operation protocol type, operation command, operation parameter, operation result status code and operation result description of the operation node. By analyzing the uplink record, the specific operation behavior that causes data quality anomalies is located.

[0112] Further, the feature offset change rate is as follows:

[0113] R=(F t -F t-1 ) / F t-1 ;

[0114] Where R represents the feature offset change rate; F t represents the feature offset of the current link; F t-1 represents the feature offset of the previous link.

[0115] The offset cumulative amount is as follows:

[0116] C m =ΣD mi ;C r =ΣD ri ;C a =Σα i ;

[0117] Where C m represents the field mapping deviation cumulative amount; D mi represents the field mapping deviation degree of the i-th link; C r represents the value range deviation cumulative amount; D ri represents the value range deviation degree of the i-th link; C a represents the credibility decay cumulative amount; α i represents the decay value of the i-th link.

[0118] For example: the first processing link reduction: 0.8505-0.8250=0.0255; the second processing link reduction: 0.8250-0.7755=0.0495; the preset confidence reduction threshold=0.04, the second processing link reduction 0.0495>0.04, marked as a high-risk attenuation point;

[0119] The first processing link feature offset F t-1 =0.05 (minor deviation); the second processing link feature offset F t =0.5+0.4992=0.9992; the rate of change R=(0.9992-0.05) / 0.05=18.98, the preset jump threshold=4 (determined by ROC curve analysis), R>4, marked as a feature offset jump point;

[0120] In combination with the high-risk attenuation point and the jump point, it is determined that the second processing link is the maximum attenuation point, and the upper chain record is retrieved: the operation command is the field C value domain adjustment, and the operation parameter is manually expanded to 1500.

[0121] S5, visual output root cause link, data blood relationship chain and feature offset change curve, traceability conclusion as audit record chaining, dynamic update of attenuation coefficient table;

[0122] Specifically, the traceability conclusion includes abnormal data identification, node information of the root cause link, feature offset analysis result, confidence change trend, operation behavior positioning result and rectification suggestion; the traceability conclusion is serialized according to the pre-defined structured format, the serialized traceability conclusion is subjected to hash operation to generate a conclusion digest, the current blockchain state information is combined to generate an audit record identification, and the complete traceability conclusion, the conclusion digest and the audit record identification are written into a new block of the blockchain; the dynamic update of the attenuation coefficient table includes that based on the operation type, the feature offset and the confidence change data in the traceability conclusion, the actual attenuation influence coefficient of each type of operation is calculated, the corresponding values in the original attenuation coefficient table are compared, when the deviation exceeds the preset adjustment threshold, the attenuation value of the corresponding operation type and complexity combination is updated; the adjustment threshold is determined by calculating the deviation distribution of the actual attenuation influence coefficient in the historical traceability conclusion and the value in the original attenuation coefficient table; in combination with the weight of the attenuation coefficient to the confidence evaluation, a hierarchical adjustment standard is set to obtain the adjustment threshold; the updated attenuation coefficient table is serialized and subjected to hash operation to generate a table digest, which is written into the blockchain together with the update timestamp and the operation node identification.

[0123] The present application provides another technical solution, a multi-source data real-time traceability audit supervision system, which comprises: a data acquisition module, a data quality verification module, a confidence evaluation module, a traceability audit module and a dynamic optimization module;

[0124] The data collection module comprises an original data collection module and a flow information collection module; the original data collection module automatically collects original feature information, performs serialization and hash operation to generate a feature digest, and generates a unique data bloodline identifier; the flow information collection module collects flow information in real time, generates a new node identifier and chains it;

[0125] The data quality verification module comprises an integrity verification module and an exception analysis module; the integrity verification module performs integrity verification on the data according to a preset period, and triggers an exception handling process when an exception is detected; the exception analysis module extracts the current feature information of the exception data, calls the corresponding complete bloodline chain, and locates the specific operation behavior that causes the exception;

[0126] The credibility evaluation module is used for initial evaluation and dynamic calculation of data credibility coefficients, and quantifies the credibility changes in the data flow process;

[0127] The traceability audit module comprises a visualization module and a record chaining module; the visualization module outputs the root cause link, data bloodline chain and feature offset change curve through a graphical interface, and intuitively presents the data exception traceability result; the record chaining module serializes and generates a hash digest of the traceability conclusion, generates an audit record identifier in combination with the blockchain state information, and chains it;

[0128] The dynamic optimization module dynamically optimizes the decay coefficient reference table based on the traceability result, and improves the accuracy of credibility evaluation.

[0129] It is apparent to those skilled in the art that the present application is not limited to the details of the foregoing exemplary embodiments, but can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and not restrictive, and the scope of the present application is defined by the appended claims rather than the foregoing description, and therefore all changes falling within the meaning and range of equivalents of the essential elements of the claims are intended to be embraced therein. Any reference signs in the claims should not be considered as limiting the claims involved.

Claims

1. A method for real-time traceability, auditing and supervision of multi-source data, characterized by: The method includes: Deploy blockchain nodes at the source of data. When data is generated, automatically collect the original feature information of the data, write it into the blockchain, generate a unique data lineage identifier, and establish a data lineage index table. When data flows, the node identification, timestamp and operation type are recorded in real time. The attenuation value is obtained from the attenuation coefficient comparison table based on the operation type and complexity. The current credibility coefficient is calculated and the flow node information, attenuation value and current credibility coefficient are added to the chain as a new node to form a chain association. Periodically check data integrity. When data quality anomalies are detected, extract the current feature information of the abnormal data, retrieve the complete lineage chain corresponding to the abnormal data, analyze the field mapping deviation and value range deviation, and associate the attenuation value of each link with the cumulative offset. The field mapping deviation is obtained by calculating the deviation between the current field structure and the original field structure. The difference includes the difference in the number of fields, the difference in field name, the difference in field type, the difference in field order, and the difference in field constraint conditions. The value range deviation is obtained by calculating the deviation between the current field value range and the original field value range. The deviation includes the upper limit offset of the value range, the lower limit offset of the value range, the value range distribution form offset, and the value range dispersion offset. Filter links where the credibility drop is greater than the threshold and mark them as high-risk attenuation points. Combine the feature offset jump points to determine the maximum attenuation point. Retrieve the chain record of the maximum attenuation point to locate the specific operation behavior. The root cause link, data lineage chain and feature offset change curve are visually output, the traceability conclusion is uploaded to the chain as an audit record, and the attenuation coefficient comparison table is dynamically updated; the traceability conclusion includes abnormal data identification, node information of the root cause link, feature offset analysis results, credibility change trend, operation behavior positioning results and rectification suggestions; the traceability conclusion is serialized according to a predefined structured format, the serialized traceability conclusion is hashed to generate a conclusion summary, and the audit record identification is generated by combining it with the current blockchain state information, and the complete traceability conclusion, conclusion summary and audit record identification are written into the new block of the blockchain; the dynamic update of the attenuation coefficient comparison table includes, based on the operation type, feature offset and credibility change data in the traceability conclusion, calculating the actual attenuation impact coefficient of each type of operation, comparing it with the corresponding value in the original attenuation coefficient comparison table, and when the deviation exceeds the preset adjustment threshold, updating the attenuation value of the corresponding operation type and complexity combination; the updated attenuation coefficient comparison table is serialized and hashed to generate a comparison table summary, which is written into the blockchain together with the update timestamp and operation node identification.

2. A method for real-time traceability, auditing and supervision of multi-source data according to claim 1, characterized in that: The original feature information includes data field structure, field value range, generation timestamp, collection device identification and initial credibility coefficient; The original feature information is serialized according to a predefined structured format, a hash operation is performed on the serialized original feature information to generate a feature digest, and the digest is combined with the current blockchain state information to generate a data lineage identifier, and the complete feature information is written into a new blockchain block; The data lineage index table includes a data lineage identifier field, an original data storage address field, and a blockchain transaction hash field; a mapping relationship between the data lineage identifier and the original data physical storage address, and an index relationship between the data lineage identifier and the blockchain transaction record are established.

3. A method for real-time traceability, auditing and supervision of multi-source data according to claim 2, characterized in that: The initial credibility coefficient includes: The initial credibility coefficient is obtained through a multi-dimensional feature evaluation model, based on a comprehensive evaluation of data type factors, data source authentication level, data integrity check value, timestamp credibility, and data generation frequency. The formula is as follows: C0=w1×DT+w2×DA+w3×DC+w4×TR+w5×DF; Among them, C0 represents the initial credibility coefficient; w1, w2, w3, w4 and w5 represent weighting coefficients; DT represents the data type factor; DA represents the data source authentication level; DC represents the data integrity check value; TR represents the timestamp credibility; DF represents the data generation frequency; The timestamp credibility is obtained based on the time difference between the data generation time and the blockchain writing time, and the formula is TR=e (-λ×Δt) , where λ represents the attenuation coefficient and Δt represents the difference between the data generation timestamp and the blockchain writing timestamp.

4. The method for real-time traceability, auditing and supervision of multi-source data according to claim 1, characterized in that: The data flow includes data collection, transmission, storage or sharing; The complexity is assessed based on the number of fields involved in the operation, the depth of the calculation logic, and the number of associated data records, and is divided into three levels: simple, medium, and complex. According to the operation type and complexity, the attenuation value is obtained from the attenuation coefficient comparison table to calculate the current credibility coefficient. The formula is as follows: C t =C t-1 ×(1-α); Among them, C t Expressed as the credibility coefficient of the current node; C t-1 It is represented as the credibility coefficient of the previous node, and α is represented as the attenuation value corresponding to the current operation; The transfer node information, attenuation value and current credibility coefficient are serialized in a predefined structured format, and a hash operation is performed on the serialized information to generate a feature summary. This is combined with the current blockchain state information to generate a new data transfer node identifier. The complete transfer node information, attenuation value, current credibility coefficient, feature summary and data transfer node identifier are written into a new block of the blockchain to form a chain association with the previous node.

5. A method for real-time traceability, auditing and supervision of multi-source data according to claim 4, characterized in that: The chain association adopts a Merkle tree structure, and each data flow node contains a pointer to its parent node. The complete data flow path can be obtained by tracing back step by step; at the same time, a data flow index table is established in the blockchain. The data flow index table includes a data lineage identifier field, a data flow node identification field, a flow timestamp field, an operation type field, an attenuation value field and a credibility coefficient field, and establishes a mapping relationship from the data lineage identifier to the data flow node, as well as a temporal association relationship between the data flow nodes.

6. The method for real-time traceability, auditing and supervision of multi-source data according to claim 1, characterized in that: The data integrity includes data field structure, field value range, data relationship constraints and data integrity constraints; When data quality anomalies are detected, the exception handling process is automatically triggered to extract the current feature information of the abnormal data, which includes the data field structure, field value range, current timestamp, current storage node identifier and current credibility coefficient; Compare the current feature information of the abnormal data with the original feature information, and analyze the field mapping deviation and value range deviation; The field mapping deviation and value range deviation are correlated with the attenuation value of each circulation link, and the offset accumulation of each link is calculated. The offset accumulation includes the field mapping deviation accumulation, the value range deviation accumulation and the credibility attenuation accumulation.

7. A method for real-time traceability, auditing and supervision of multi-source data according to claim 6, characterized in that: Using the time series analysis algorithm, the attenuation value, characteristic offset and cumulative offset of each link are analyzed in time series correlation to identify the changing trend, mutation point and periodic fluctuation pattern of characteristic offset; By comparing the credibility drop of each link with the preset threshold, the links with a credibility drop greater than the threshold are screened out and marked as high-risk attenuation points; Determine the attenuation point with the greatest impact on data quality based on the location and amplitude of the feature offset jump point. The feature offset jump point is obtained by calculating the rate of change of the feature offset of adjacent links. Points where the rate of change exceeds a preset jump threshold are marked as feature offset jump points. Retrieve the on-chain record of the maximum attenuation point, where the on-chain record includes the physical address, logical address, device identifier, system identifier, user identifier, operation IP address, operation port number, operation protocol type, operation command, operation parameters, operation result status code, and operation result description of the operation node; and locate the specific operation behavior that causes data quality abnormality by analyzing the on-chain record.

8. A method for real-time traceability, auditing and supervision of multi-source data according to claim 7, characterized in that: The field mapping deviation is as follows: D m =(n1+n2+n3+n4+n5) / N; Among them, D m It represents the field mapping deviation; n1 represents the difference in the number of fields; n2 represents the difference in the field name; n3 represents the difference in the field type; n4 represents the difference in the field order; n5 represents the difference in the field constraint; N represents the total number of original fields; The value range deviation is as follows: D r =(|U c -U o |+|L c -L o |+|F c -F o |+|S c -S o |) / (U o +L o +F o +S o ); Among them, D r Expressed as the deviation of the value range; U c Indicates the upper limit of the current field value range; U o Indicates the upper limit of the original field value range; L c Indicates the lower limit of the current field value range; L o Indicates the lower limit of the original field value range; F c Represents the distribution morphological parameters of the current field value range; F o Expressed as the original field value range distribution morphological parameter; S c It is expressed as the discrete degree parameter of the current field value range; S o Expressed as the discrete degree parameter of the original field value range; The characteristic offset change rate is as follows: R=(F t -F t-1 ) / F t-1 ; Among them, R represents the rate of change of characteristic offset; F t Expressed as the characteristic offset of the current link; F t-1 Expressed as the feature offset of the previous link; The cumulative offset is calculated as follows: C m =ΣD mi ;C r =ΣD ri ;C a =Sa i ; Among them, C m Expressed as the cumulative amount of field mapping deviation; D mi It is expressed as the field mapping deviation of the i-th link; C r Expressed as the cumulative deviation from the value range; D ri Expressed as the value range deviation of the i-th link; C a Expressed as the cumulative amount of credibility decay; α i Expressed as the attenuation value of the i-th link.

9. A system for real-time traceability, auditing, and supervision of multi-source data, applied to a method for real-time traceability, auditing, and supervision of multi-source data according to any one of claims 1 to 8, characterized in that: The system includes: data acquisition module, data quality verification module, credibility assessment module, traceability audit module and dynamic optimization module; The data acquisition module includes an original data acquisition module and a flow information acquisition module; the original data acquisition module automatically collects original feature information, performs serialization and hash operations to generate feature summaries, and generates unique data lineage identifiers; the flow information acquisition module collects flow information in real time, generates new node identifiers, and uploads them to the chain; The data quality verification module includes an integrity verification module and an anomaly analysis module; the integrity verification module performs integrity verification on the data according to a preset period and triggers the anomaly handling process when an anomaly is detected; the anomaly analysis module extracts the current feature information of the abnormal data, retrieves the corresponding complete lineage chain, and locates the specific operation behavior that caused the anomaly; The credibility assessment module is used for the initial assessment and dynamic calculation of the data credibility coefficient and quantifies the credibility changes during the data flow process; The traceability audit module includes a visualization module and a record-uploading module. The visualization module outputs the root cause link, data lineage chain, and feature offset change curve through a graphical interface, intuitively presenting the data anomaly traceability results. The record-uploading module serializes the traceability conclusion and generates a hash summary, which is combined with the blockchain status information to generate an audit record identifier and then uploaded to the chain. The dynamic optimization module dynamically optimizes the attenuation coefficient comparison table based on the traceability results to improve the accuracy of the credibility assessment.

Images