A port multi-source business data quality governance method and system

Abstract

The application discloses a kind of port multi-source business data quality governance method and system, it is related to data processing technical field.The kind of port multi-source business data quality governance method and system, comprising the following steps: S1, obtain original governance data, and carry out time axis reposition, structure normalization and encapsulation processing;S2, extract dimension candidate record, evaluate the freezing credibility of candidate dimension version;S3, execute candidate version recall, build time state association candidate set, evaluate time state mismatch degree;S4, the release state of current release cycle is evaluated, and the release state of current result is identified.The application effectively improves consistency, stability and traceability in cross-system convergence, solves the problem that dimension table version change and fact record processing are not synchronized, which is prone to produce silent mismatch, mismatch result is difficult to block in time, and abnormal result is easy to spread pollution downstream.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for managing the quality of multi-source business data in ports. Background Technology

[0002] With the development of port digital management and control platforms, data middleware, data services, and intelligent analysis applications, the scale of multi-source heterogeneous data access continues to expand, the frequency of data version switching is constantly increasing, and the cross-layer data association and result reuse links are becoming increasingly complex. In existing technologies, current systems typically use static dimension mapping and ordinary key-value association to complete data governance. This lack of coordinated control over event time, effective intervals, version boundaries, semantic fingerprints, and release gates easily leads to problems such as superficially successful associations but actual semantic deviations, incorrect result releases after version switching, and the continuous spread of abnormal results. When dimension table changes and fact table processing are not synchronized, the Join mechanism can produce silent mismatches. The platform often has no alerts, but the results have already shifted; it's not that the source data is corrupted, but rather that the code table, mapping table, and standard table have been updated at some point, while the fact table is still calculated according to the old dimension meaning; or the fact table is re-run and a new dimension version is obtained. Most existing platform Joins only look at whether the primary key can be successfully associated, without seeing which time segment dimension version is being matched during the association. Therefore, the results appear complete on the surface, but the semantics are actually skewed.

[0003] Therefore, in order to address the above issues, there is an urgent need for a method and system for the quality governance of multi-source business data in ports. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for the quality governance of multi-source business data in ports, which solves the problems of silent mismatches, difficulty in timely blocking of mismatch results, and easy spread and contamination of abnormal results downstream when dimension table version changes and fact record processing are out of sync.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for quality governance of multi-source business data in ports, comprising the following steps:

[0006] S1. Obtain the raw governance data and perform timeline relocation, structure normalization, and encapsulation processing on the raw governance data to obtain the temporal raw dataset; S2. Extract candidate records of dimensions based on the temporal raw dataset, generate candidate dimension version sequences, evaluate the credibility of freezing candidate dimension versions, and construct a frozen dimension snapshot library; S3. Based on the frozen dimension snapshot library, perform candidate version recall, construct a temporal association candidate set, evaluate the degree of temporal mismatch, and form a temporal association dataset; S4. Based on the temporal association dataset, evaluate the release status of the current release cycle, identify the release status of the current result, and generate channel-level response behavior labels.

[0007] Further, the specific process of acquiring raw governance data and performing timeline relocation, structural normalization, and encapsulation on the raw governance data to obtain the temporal raw dataset is as follows: Raw governance data is acquired by collecting database log streams, interface return streams, message queue streams, file import streams, and cache flushing streams from heterogeneous data links. The raw governance data includes object identifier keys, source record payloads, source-side event times, acquisition times, governance platform entry times, source location identifiers, field structure information, and original content summary information. A timeline relocation method based on three-timescale alignment is used to uniformly correct the source-side event times, acquisition times, and governance platform entry times in the raw governance data. A structural normalization method based on field signature mapping is used to uniformly process the field names, field types, null value encodings, and time formats in the raw governance data to form standard record units. A candidate labeling method based on record role discrimination is used to classify the standard record units into fact candidate records, dimension candidate records, or correction candidate records. The standard record units after type classification are then temporally encapsulated to generate the temporal raw dataset.

[0008] Furthermore, the specific process of extracting candidate dimension records and generating candidate dimension version sequences based on the temporal raw dataset is as follows: Based on the temporal raw dataset, standard record units with record role identifiers as candidate dimension records are selected; candidate dimension records are merged according to object identifier keys to form an object-level candidate version set; candidate dimension records under the same object identifier key are arranged in chronological order of events, and continuous records are version-segmented to generate candidate dimension version sequences; adjacency associations are established for adjacent versions in the candidate dimension version sequence, conflict associations are established for candidate dimension versions with overlapping time intervals, and evolutionary associations are established for candidate dimension versions with rollback, replacement, or parallel update relationships to construct a dimension version graph; the duration is obtained by comparing the candidate dimension version sequences arranged in chronological order under the same dimension object; and the duration is obtained by comparing the same type of dimension... The freeze time constant is obtained by statistically analyzing the duration of adjacent candidate versions of a dimension object; the field state vector is obtained by concatenating the field value encoding, field non-empty bitmap, field type signature, and field domain distribution parameters corresponding to the candidate dimension version; the state difference is obtained by calculating the L2 norm after subtracting the field state vector of the current candidate version from the field state vector of the previous candidate version; the field state change scale benchmark is obtained by statistically analyzing the state difference between adjacent candidate versions of the same dimension object or the same type of dimension object; the adjacent version set is obtained by extracting the previous and next adjacent candidate dimension versions within the evolution window range under the same dimension object, with the event time of the current candidate version as the center; the interval overlap rate is obtained by calculating the intersection length of the effective interval of the current candidate version and the effective interval of the adjacent version, and then dividing it by the sum of the union length of the two and the zero-prevention term.

[0009] Furthermore, the credibility of freezing candidate dimension versions is evaluated. The specific process of constructing the frozen dimension snapshot library is as follows: Divide the duration by the sum of the freezing time constant and the zero-prevention term, and input it into the hyperbolic tangent function to obtain the duration contribution term; divide the state difference by the sum of the field state change scale benchmark and the zero-prevention term, take the negative and perform an exponential transformation to obtain the difference suppression term; subtract one from the interval overlap rate between the current candidate version and each adjacent version, and then multiply the results sequentially to obtain the overlap constraint term; multiply the duration contribution term, the difference suppression term, and the overlap constraint term sequentially to obtain the snapshot freezing credibility value; compare the snapshot freezing credibility value with the snapshot freezing threshold in real time; when the snapshot freezing credibility value is greater than or equal to the snapshot freezing threshold, the current candidate version is written into the frozen dimension snapshot library and marked as a formally frozen version; when the snapshot freezing credibility value is lower than the snapshot freezing threshold, the current candidate version is kept in the waiting area for freezing, and will be judged again after the next round of evaluation; organize the selected formally frozen versions by dimension object key, effective start time, and effective end time to construct the frozen dimension snapshot library.

[0010] Furthermore, based on the frozen dimension snapshot library, the specific process of performing candidate version recall and constructing a temporal association candidate set is as follows: Based on the original temporal dataset, standard record units with record role identifiers as fact candidate records are extracted; after consistency correction of the event time, collection time, and entry time of the fact candidate records in the original temporal dataset, a time window is reconstructed by combining the time interval of adjacent fact candidate records under the same object identifier key to obtain a reference time interval; the effective interval is obtained by locating the version boundaries of the candidate dimension version sequence arranged in the order of event time under the same dimension object; the intersection length and union length are obtained by calculating the interval intersection and union of the reference time interval and the effective interval; and field semantic encoding and frequency normalization are performed on the fact candidate records and candidate dimension versions respectively. The field names and values ​​of each record are extracted to form a word sequence. The frequency of each word is statistically analyzed based on the global bag-of-words model and normalized to a probability distribution to obtain the semantic fingerprint probability vector of each record. The Jensen-Shannon divergence is obtained by calculating the symmetric divergence based on the intermediate distribution of the two. The reference time base is obtained by projecting the event time, collection time and entry time of the candidate fact records after ranking them by credibility. The center time is obtained by finding the median of the start time and end time of the effective interval of the candidate dimension version. The time offset base constant is obtained by statistically analyzing the absolute offset between the reference time of the candidate fact record and the center time of the hit dimension version in the associated samples. The candidate dimension version set with the same object identifier key is recalled in the frozen dimension snapshot library according to the object identifier key to form the temporal association candidate set.

[0011] Furthermore, the specific process for assessing the degree of temporal mismatch and forming a temporal association dataset is as follows: Add a zero-prevention term to the intersection length, then divide by the union length plus the zero-prevention term to obtain the time interval overlap ratio; take the natural logarithm of the time interval overlap ratio and then take the negative value to obtain the time interval mismatch term; use the Jensen-Shannon divergence as the semantic difference term; calculate the absolute difference between the reference time base and the center time, divide the absolute difference by the sum of the time offset base constant and the zero-prevention term, then add one and take the natural logarithm to obtain the time center offset term; combine the time interval mismatch term, semantic difference term, and time center offset term... The three offset terms are added together to obtain the temporal join mismatch energy; the temporal join mismatch energy is compared with the mismatch threshold in real time; when the temporal join mismatch energy is less than the mismatch threshold, the current hit version is confirmed as a valid associated version, and the current fact candidate record is formally temporally associated with the valid associated version; when the temporal join mismatch energy is greater than or equal to the mismatch threshold, the current fact candidate record is marked as a silent mismatch record and written into the mismatch buffer; after completing the above temporal association for all fact candidate records, a temporal association dataset and a silent mismatch tag set are formed, providing input for result release control and replay repair.

[0012] Furthermore, based on the temporal correlation dataset, the specific process for evaluating the release status of the current release cycle is as follows: Based on the temporal correlation dataset, the version consistency rate is obtained by statistically analyzing the proportion of version-consistent results to the total number of results; the number of silent mismatch records is obtained by counting the records that enter the silent mismatch marker set within the current release cycle; the influence radius is obtained by tracking and statistically analyzing the downstream result nodes associated with the silent mismatch records in the current release cycle; the number of corrected results is obtained by statistically analyzing the number of records that have completed isolation repair, version replay correction, or regenerated candidate release results within the current release cycle; and the previous release... The gate self-calibration value is obtained by exponentially calculating the gate inertia coefficient. The gate retention term is obtained by exponentially calculating the sum of the version consistency rate and the zero prevention term by subtracting the gate inertia coefficient from 1. The consistency contribution term is obtained by exponentially calculating the sum of the number of silent mismatch records by one and taking the natural logarithm. The mismatch amplification term is obtained by exponentially calculating the influence radius by one. The mismatch amplification term and the influence amplification term are multiplied together, divided by one, and the number of calibration results is added to obtain the mismatch suppression term. The mismatch suppression term is negative and then exponentially transformed to obtain the risk attenuation term. The gate retention term, consistency contribution term, and risk attenuation term are multiplied sequentially to obtain the release gate self-calibration value.

[0013] Furthermore, the specific process for identifying the current release status of the result is as follows: Based on the release gate self-correction value of the current release period, the release status of the current release period is identified; when the release gate self-correction value is not lower than the release gate threshold, the current result is determined to meet the release conditions; when the release gate self-correction value is lower than the release gate threshold, the current result is determined not to meet the release conditions; when the current result is determined to meet the release conditions, the current set of results to be released is written into the formal release layer; when the current result is determined not to meet the release conditions, the current set of results to be released is written into the isolation buffer, and a freeze flag is set for the affected result nodes, pausing the continued transmission to the downstream service layer; at the same time, version replay is performed on the results in the isolation buffer based on the latest frozen dimension snapshot, and candidate release results are regenerated.

[0014] Furthermore, the specific process for generating channel-level response behavior tags is as follows: the current release status, the number of silent mismatch records, the radius of influence, the number of correction results, the number of replays, and the average mismatch energy after replay are structurally bound to generate release governance tags; the release governance tags include: release cycle number, release status, number of mismatches, radius of influence, number of repairs, replay flag, and current rule version number; when a continuous release cycle is in a prohibited release state, or the radius of influence exceeds the radius of influence threshold, the rule review process is triggered to recalibrate the freeze threshold, mismatch threshold, and release gate threshold; the release governance tags, isolation records, and replay results formed in the current cycle are summarized to form a governance closed-loop record set.

[0015] The second aspect of this invention provides a port multi-source business data quality governance system, applied to a port multi-source business data quality governance method, comprising: a data acquisition and temporal encapsulation module, used to acquire raw governance data and perform time axis relocation, structure normalization, and encapsulation processing on the raw governance data to obtain a temporal raw dataset; a dimension snapshot freezing module, used to extract dimension candidate records based on the temporal raw dataset, generate a candidate dimension version sequence, evaluate the freezing credibility of the candidate dimension versions, and construct a frozen dimension snapshot library; a temporal association verification module, used to perform candidate version recall based on the frozen dimension snapshot library, construct a temporal association candidate set, evaluate the temporal mismatch degree, and form a temporal association dataset; and a release closed-loop control module, used to evaluate the release status of the current release cycle based on the temporal association dataset, identify the release status of the current result, and generate channel-level response behavior labels.

[0016] The present invention has the following beneficial effects:

[0017] (1) This invention improves the alignability and manageability of heterogeneous data from the source by performing time axis relocation, structure normalization, role labeling and temporal encapsulation on the original governance data during the collection stage. It can establish joint constraints between version stability, adjacent difference degree and interval overlap degree, and prevent dimension versions that are not yet stable or have interval conflicts from entering the formal snapshot layer, thereby improving the stability and availability of the dimension version library.

[0018] (2) In this invention, temporal association candidate pairs are constructed based on reference time interval, effective interval, semantic fingerprint probability vector and center time offset, and temporal join mismatch energy is used to perform optimal hit screening of candidate dimension versions. This makes temporal association judgment no longer dependent on single primary key hit or single time comparison, but forms a comprehensive verification mechanism oriented towards version, semantic and temporal three-dimensional constraints, thereby enhancing the system's ability to identify hidden quality problems.

[0019] (3) This invention dynamically controls the release of formal results based on the self-correction value of the release gate, and recursively updates the data by combining version consistency rate, number of silent mismatch records, influence radius, and number of correction results. This allows for automatic tightening of releases when the risk of mismatch propagation increases, and automatic recovery of release capabilities when the number of correction results increases, thereby improving the stability and adaptive control capabilities of the result release process. It completes result reconstruction and correction without directly polluting the formal release layer, preventing abnormal results from directly entering downstream service links, and improving the system's isolation and self-recovery capabilities for abnormal results.

[0020] (4) This invention, by structurally binding the number of silent mismatch records, the radius of influence, the number of repairs, the playback results and the current rule version, forms a set of release governance tags and governance closed-loop records, which can provide clear basis for subsequent quality traceability, threshold re-inspection, rule reuse and version playback, and improve the interpretability and auditability of the data governance process.

[0021] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0022] Figure 1 This is a flowchart of a port multi-source business data quality governance method according to the present invention;

[0023] Figure 2 This is an architecture diagram of a port multi-source business data quality governance system according to the present invention;

[0024] Figure 3 This is a scatter bubble chart of the candidate dimension version freeze determination for this invention.

[0025] Figure 4 This is the energy thermogram of temporal join mismatch in this invention;

[0026] Figure 5 This invention presents a joint evolution diagram of the periodic gate value and the radius of influence. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Please see Figures 1-5 This invention provides a technical solution: a method for quality governance of multi-source business data in ports, comprising the following steps: S1, acquiring raw governance data, and performing time axis relocation, structure normalization, and encapsulation processing on the raw governance data to obtain a temporal raw dataset; S2, extracting candidate records of dimensions based on the temporal raw dataset, generating a candidate dimension version sequence, evaluating the freezing credibility of the candidate dimension versions, and constructing a frozen dimension snapshot library; S3, based on the frozen dimension snapshot library, performing candidate version recall, constructing a temporal association candidate set, evaluating the degree of temporal mismatch, and forming a temporal association dataset; S4, based on the temporal association dataset, evaluating the release status of the current release cycle, identifying the release status of the current result, and generating channel-level response behavior labels.

[0029] Specifically, the process of obtaining raw governance data and performing timeline relocation, structure normalization, and encapsulation on the raw governance data to obtain the temporal raw dataset is as follows: the raw governance data is obtained by collecting database log streams, interface return streams, message queue streams, file import streams, and cache flushing streams in the heterogeneous data link; the raw governance data includes object identification keys, source record payloads, source side event times, collection times, governance platform entry times, source site identifiers, field structure information, and original content summary information.

[0030] By using a timeline relocation method based on three timescale alignment, the source-side event time, acquisition time, and governance platform storage time in the original governance data are uniformly corrected. The source-side event time is used as the primary timeline benchmark to characterize the actual time when the business occurs; the acquisition time is used as the secondary timeline benchmark to compensate for the initial delay of the transmission link; and the governance platform storage time is used as the drift compensation benchmark to identify and correct timing misalignment issues caused by link delays, replay arrivals, supplementary acquisition and re-feedback, and asynchronous writing, thereby reducing version mismatch issues caused by data out-of-order or transmission delays.

[0031] By using a structure normalization method based on field signature mapping, the field names, field types, null value encoding, and time formats in the original governance data are uniformly processed. The field signature includes a combined hash of the field name, field type, value range constraints, and null value encoding method. Based on this field signature, fields with the same semantics but different physical representations in heterogeneous data sources are mapped to a unified field identifier. The field names are standardized by aliasing, the field types are converted for compatibility, the null value encoding is unified to NULL or a default value is specified, and the time format is unified to ISO-8601 or a timestamp format with specified precision, forming a standard record unit.

[0032] The standard record unit is classified into different types based on the candidate labeling method that determines the record role. Specifically, the standard record unit is labeled as a fact candidate record, dimension candidate record, or correction candidate record based on whether there is a version update identifier, dimension change identifier, or correction compensation identifier in the record, and in combination with whether the difference between the source event time and the collection time exceeds the late threshold, as well as the degree of matching between the field signature and the baseline signature. Fact candidate records are used to participate in temporal correlation, dimension candidate records are used to participate in version freezing, and correction candidate records are used to participate in playback correction and result repair.

[0033] After the standard record units have been classified into types, they are temporally encapsulated to generate a temporal raw dataset. Each record in the temporal raw dataset includes at least: object identifier key, source record payload, event time, acquisition time, storage time, source location identifier, structure summary value, content summary value, and record role identifier. At the same time, late records, duplicate records, and structure drift records are marked with anomaly flags and written to the late record buffer, duplicate record buffer, and structure check queue, respectively, to provide a unified temporal basis for dimension version freezing, temporal join verification, and release gate control.

[0034] This implementation scheme effectively reduces version mismatch issues caused by data out-of-order or transmission delays through joint correction using three timescales, improving temporal consistency. Field signature mapping achieves semantic uniformity and format compatibility across heterogeneous data sources, eliminating structural heterogeneity. Record role identification accurately identifies facts, dimensions, and correction candidate records, providing a clear data classification foundation for temporal correlation, version freezing, and replay correction. Simultaneously, late, duplicate, and structurally drifting records are marked with anomaly tags and written to their corresponding buffers, ensuring that abnormal data is isolated and does not contaminate the main process. This results in a unified temporal dataset with a unified structure and clear roles, providing a high-quality, traceable temporal data foundation for dimension version freezing, temporal correlation verification, and release gate control.

[0035] Specifically, the process of extracting candidate dimension records and generating a candidate dimension version sequence based on the temporal raw dataset is as follows: Based on the temporal raw dataset, standard record units with record role identifiers as candidate dimension records are selected; candidate dimension records are merged according to object identifier keys to form an object-level candidate version set; candidate dimension records under the same object identifier key are arranged in chronological order of events, and consecutive records are version-segmented to generate a candidate dimension version sequence; each candidate dimension version includes at least: dimension object key, version summary value, effective start time, effective end time, frozen status flag, version confidence coefficient, and field state vector. The field state vector is composed of field value encoding, field non-empty bitmap, field type signature, and field domain distribution parameters concatenated together. The concatenation of field state vectors is used to compress the field-level structural features and value distribution features of the dimension records into a fixed-length comparable vector, which facilitates subsequent calculation of state differences and version evolution degree; the reason for using concatenation instead of weighted summation is to preserve the independent contribution of each field and avoid information aliasing. The field distribution parameters are in the form of a lightweight statistical summary, including the mean, variance, minimum and maximum values ​​of the field values, and histogram bin counts of equal-frequency bins. For string types, length distribution and first character frequency distribution are used. Full embedding is not used to maintain interpretability and computational efficiency.

[0036] The field state vector is constructed by concatenating field value encoding, field non-empty bitmap, field type signature, and field domain distribution parameters. Adjacency associations are established for adjacent versions in the candidate dimension version sequence; conflict associations are established for candidate dimension versions with overlapping time intervals; and evolutionary associations are established for candidate dimension versions with rollback, replacement, or parallel update relationships, thus constructing a dimension version graph. The selection rules for the adjacent version set are as follows: In the dimension version graph, for the current candidate version, the direct successor and successor versions are first obtained through evolutionary association edges. If no evolutionary association edge exists, it degenerates into temporal adjacency ordered by event time. Simultaneously, versions connected by conflicting association edges are not included in the adjacent version set to avoid conflicting versions interfering with state difference calculations. When the adjacent version set is used to calculate state difference and interval overlap rate, it only includes versions in the evolutionary and temporal adjacent versions that are not marked with conflicting associations.

[0037] The duration is obtained by comparing candidate dimension version sequences arranged chronologically for the same dimension object; the freeze time constant is obtained by statistically analyzing the durations of adjacent candidate versions of the same type of dimension object; the freeze time constant is the median duration of adjacent candidate versions of the same type of dimension object, with a statistical window period of the past 7 days or at least 100 version events, updated every 24 hours. The field state vector is obtained by concatenating the field value encoding, field non-empty bitmap, field type signature, and field domain distribution parameters corresponding to the candidate dimension version.

[0038] The state difference is obtained by calculating the L2 norm after subtracting the field state vector of the current candidate version from that of the previous candidate version. Using the L2 norm to calculate the state difference can comprehensively measure the overall change of the field state vector in each dimension. It is smoother than the L1 norm, more sensitive to moderate changes than the infinite norm, and has a certain suppression effect on noise. It is suitable for judging whether the dimension version has undergone substantial changes.

[0039] The baseline for field state change is obtained by statistically analyzing the state differences between adjacent candidate versions of the same dimension object or similar dimension objects. Within the same dimension object, adjacent candidate dimension versions are extracted within the evolution window, centered on the event time of the current candidate version, to obtain the set of adjacent versions. The evolution window range is determined adaptively: K candidate versions before and after the event time of the current version are selected, with K defaulting to 3, and the window time span not exceeding twice the historical average duration of the same object; if there are insufficient versions before and after, all available versions are selected. The interval overlap rate is obtained by calculating the intersection length of the effective interval of the current candidate version and the effective interval of the adjacent versions, and then dividing it by the sum of their union length and the zero-prevention term. The interval overlap rate quantifies the degree of overlap between two candidate dimension versions on the time axis. The intersection-union ratio is a dimensionless normalized indicator that eliminates the influence of absolute time length, facilitating the uniform setting of overlap thresholds across different objects. When the overlap rate exceeds the preset threshold, it is determined to be a conflicting version, used to trigger conflict resolution or version merging. The zero-prevention term ε is a constant, taking the value of the smallest time granularity to ensure it has the same dimension as the union length and avoid division-by-zero errors.

[0040] By concatenating field state vectors and using the L2 norm difference, we can quantitatively identify substantial changes between versions and avoid accidental freezing due to irrelevant field fluctuations. By constraining conflicting versions through interval overlap rate and combining the statistical caliber of the freeze time constant, compared with the freeze method that only uses duration, we reduce the probability of conflicting versions being mistakenly added to the database and improve the stability and consistency of the dimensional version sequence.

[0041] In this implementation scheme, the design of splicing field state vectors effectively preserves the structural and value distribution characteristics of the dimensional record fields. By combining the L2 norm to calculate the state difference, the substantive changes between dimensional versions can be accurately and quantitatively identified, avoiding false freezes caused by irrelevant field fluctuations. By constructing a dimensional version graph and constraining the adjacent version set, and combining the interval overlap rate to quantify the degree of version time overlap, conflicting versions can be effectively identified, reducing the probability of conflicting versions being mistakenly entered into the database. At the same time, based on the freeze time constant obtained from the statistics of similar dimensional objects, the stability and consistency of the dimensional version sequence are further improved. Overall, the accuracy and efficiency of dimensional version extraction, sequence generation, and conflict identification are optimized, taking into account both computational efficiency and interpretability, and providing reliable support for dimensional version management and related data processing.

[0042] Specifically, the process of evaluating the freeze credibility of candidate dimension versions and constructing a frozen dimension snapshot library is as follows: Divide the duration by the sum of the freeze time constant and the zero-prevention term, then input the result into a hyperbolic tangent function to obtain the duration contribution term; divide the state difference by the sum of the field state change scale benchmark and the zero-prevention term, take the negative, and perform an exponential transformation to obtain the difference suppression term; subtract one from the interval overlap rate between the current candidate version and each adjacent version, then multiply the results sequentially to obtain the overlap constraint term; multiply the duration contribution term, difference suppression term, and overlap constraint term sequentially to obtain the snapshot freeze credibility value. The duration contribution term, difference suppression term, and overlap constraint term are combined using a multiplicative structure, rather than an additive weighted sum. The technical effect is that if any dimension is abnormal, the corresponding single term can significantly reduce the overall snapshot freeze credibility value, even to zero, thereby effectively suppressing the erroneous freezing of versions due to brief fluctuations, conflicting versions, or overlapping versions. The multiplicative structure inherently possesses a veto characteristic, which, compared to an additive structure, can more reliably guarantee the quality and stability of the frozen versions.

[0043] The specific formula for calculating the snapshot freeze confidence value is as follows:

[0044] ;

[0045] In the formula, Indicates the first The first dimension object The snapshot freeze confidence value of each candidate version is used to characterize whether the current candidate version has reached the state where it can be frozen and written to the formal snapshot library; This indicates the duration of a candidate version since the most recent change in the field's state, representing the length of time the current candidate version has remained unchanged. This represents the freeze time constant, with a value greater than zero. It is used to normalize the duration of sustained stability, making the stability of different objects and different update frequencies comparable. Indicates the first Each candidate version has a field state vector, which is used to characterize the overall field state features of the current candidate version. This represents the field state vector of the previous candidate version, which is used as a direct comparison benchmark for the current candidate version. This represents the state difference between the current candidate version and the previous candidate version, and is used to characterize the overall change of the current candidate version relative to the previous candidate version. This represents the benchmark for the scale of field state changes, used to normalize the scale of the current state difference. This represents the set of adjacent versions within the same object evolution window as the current candidate version, used to limit the range of comparison objects for interval conflict determination with the current candidate version; This represents the interval overlap rate between the current candidate version and its adjacent versions, used to characterize the degree of overlap between the current candidate version and its adjacent versions over a time interval; This indicates the zero-prevention term, obtained by setting a small positive number, with a value range of 10. 8 Up to 10 3 This is used to avoid situations where the denominator is zero when the freeze time constant, field state change scale benchmark, or interval union length is zero.

[0046] The system compares the snapshot freeze confidence value with the snapshot freeze threshold in real time. When the snapshot freeze confidence value is greater than or equal to the snapshot freeze threshold, the current candidate version is written to the frozen dimension snapshot library and marked as a formally frozen version. The current system time is appended as a freeze timestamp during writing, and the specific values ​​of the duration contribution, difference suppression, and overlap constraint items that triggered the freeze are recorded. When the snapshot freeze confidence value is lower than the snapshot freeze threshold, the current candidate version is kept in the waiting-to-freeze area, awaiting further evaluation in the next round. The waiting-to-freeze area is organized by object identifier key and entry time. The snapshot freeze confidence value is recalculated in each evaluation cycle, and a counter for consecutive failed rounds is accumulated. The selected formally frozen versions are indexed and organized by dimension object key, effective start time, and effective end time to build a frozen dimension snapshot library. The index uses a B+ tree structure, supporting efficient retrieval by object key and time range, and simultaneously saving snapshot copies of the version summary value and field status vector for each formally frozen version.

[0047] like Figure 3The scatter bubble chart showing the candidate dimension version freeze determination illustrates the distribution of different candidate dimension versions across two dimensions: duration of sustained stability and state difference, along with their corresponding snapshot freeze confidence values. The horizontal axis represents duration of sustained stability, and the vertical axis represents state difference. The version numbers in the bubble labels identify different candidate dimension versions, and the values ​​below the labels represent the snapshot freeze confidence values ​​for the corresponding candidate dimension versions. The vertical dashed lines in the chart are reference lines for stability duration, and the horizontal dashed lines are reference lines for state difference. As can be seen from the chart, candidate dimension versions located in the lower right region, such as V3, V5, and V6, have longer durations of sustained stability and smaller state differences, resulting in relatively higher snapshot freeze confidence values. This indicates that these candidate dimension versions remain stable over time and have minimal differences from the previous version, making them more suitable for inclusion in the frozen dimension snapshot library as official snapshots. Conversely, candidate dimension versions located in the upper left region, such as V2, V4, and V7, have shorter durations of sustained stability and larger state differences, resulting in relatively lower snapshot freeze confidence values. This indicates that these candidate dimension versions still exhibit significant fluctuations or jumps and are not suitable for direct inclusion in the official snapshot layer. Figure 3 V1 is located near the reference line, indicating that its stability and degree of difference are in an intermediate state, and the corresponding snapshot freeze confidence value is also near the critical point. Overall, this figure intuitively reflects the technical approach of this invention, which determines the freeze of candidate dimension versions by simultaneously examining the continuous stability of candidate dimension versions and the degree of difference between adjacent versions. This approach can effectively filter out unstable versions or versions with high conflict risks, thereby improving the stability and reliability of the frozen dimension snapshot library.

[0048] This implementation plan effectively suppresses the accidental freezing of unqualified candidate versions by scientifically quantifying the confidence value of snapshot freezing and combining it with the veto characteristic of the multiplicative combination evaluation model, thus ensuring the quality of frozen versions. Through dynamic screening and cyclic evaluation mechanisms, versions that are consistently stable, have reasonable differences, and have no obvious conflicts are selected, improving the consistency and reliability of the frozen dimension snapshot library. With the help of an efficient index structure and snapshot copy retention, the efficiency of library retrieval is ensured, providing support for version traceability and status review. At the same time, through visualization to assist in parameter optimization, the overall accuracy, standardization, and efficiency of dimension version freezing management are improved, laying a solid foundation for dimension data application management.

[0049] Specifically, the process of constructing a temporally related candidate set based on the frozen dimension snapshot library is as follows: Based on the original temporal dataset, standard record units with record role identifiers as fact candidate records are extracted; after consistency correction of the event time, collection time, and storage time of the fact candidate records in the original temporal dataset, a time window is reconstructed by combining the time intervals of adjacent fact candidate records under the same object identifier key to obtain a reference time interval; the effective interval is obtained by locating the version boundaries of the candidate dimension version sequence arranged in event time order under the same dimension object; the intersection length and union length are obtained by calculating the intersection and union of the reference time interval and the effective interval; and so on. Semantic field encoding and frequency normalization are performed on the fact candidate records and candidate dimension versions respectively: the field names and values ​​in each record are extracted, and the field names and values ​​are treated as independent tokens to form a token sequence; the frequency of each token in the entire dataset is counted based on the global bag-of-words model, and the token frequency is normalized to a probability distribution to obtain their respective semantic fingerprint probability vectors; this method does not use TF-IDF or neural network embedding to maintain the interpretability and lightweight nature of the bag-of-words probability distribution; the field alignment strategy between the fact candidate records and candidate dimension versions is as follows: only tokens with the same field name are jointly counted, and tokens with different field names are counted independently into their respective vectors but do not participate in the divergence calculation between the two.

[0050] The Jensen-Shannon divergence is obtained by calculating the symmetric divergence based on the intermediate distribution of the two. The Jensen-Shannon divergence is used to measure the similarity between two probability distributions, with a value ranging from 0 to 1. The smaller the value, the closer the semantics. A reference time benchmark is obtained by projecting the event time, collection time, and entry time of the candidate fact records into the database after ranking them by credibility. The credibility ranking is based on the preset weights of the reliability of each timestamp: the source side event time has the highest credibility, followed by the collection and capture time, and the governance platform entry time has the lowest credibility. The projection operation is a weighted average based on credibility weights, projecting the three timestamps onto a reference point on a unified time axis. If a timestamp is missing, it is ignored and the weights are renormalized.

[0051] The center time is obtained by finding the median of the start and end times of the effective interval of the candidate dimension version; the time offset benchmark constant is obtained by statistically analyzing the absolute offset between the reference time of the fact candidate record in the associated sample and the center time of the hit dimension version; the statistical method is: within the historical window, the median of the absolute time difference between all successfully associated fact candidate records and the corresponding dimension version is taken as the time offset benchmark constant to reduce the interference of abnormal outliers.

[0052] Discrete encoding is performed on the field names, field types, field value ranges, field length ranges, null values, enumeration states, field order positions, and source role identifiers in the current fact candidate records to generate field-level semantic encoding units. All field-level semantic encoding units are mapped to a preset semantic primitive dictionary, and the occurrence frequency of each semantic primitive in the current fact candidate records is counted to obtain a semantic primitive counting sequence. The semantic primitive counting sequence is then normalized to generate a semantic fingerprint probability vector of the current fact candidate records. The semantic fingerprint probability vector is used to characterize the probabilistic features of field combinations, field distributions, and summary patterns.

[0053] Based on the object identifier key, a set of candidate dimension versions with the same object identifier key is recalled from the frozen dimension snapshot library to form a temporal association candidate set. For each candidate dimension version, the relocation event time of the first record of the current candidate dimension version is determined as the effective start time, and the relocation event time of the first record of the next candidate dimension version is determined as the effective end boundary, using an interval extraction method based on version boundary positioning. If there is no next candidate dimension version, the current release truncation time is determined as the effective end boundary. The center time of the effective interval is obtained by finding the median of the effective start time and the effective end boundary. The field name, field type, field value mode, null value status, and field position in the current candidate dimension version are encoded and statistically analyzed using a method based on field semantic encoding and frequency normalization to generate a dimension version semantic fingerprint probability vector. Then, a three-level pairing method based on object identifier key initial screening, reference time interval and effective interval overlap verification secondary screening, and semantic fingerprint similarity comparison is used to pair the effective interval, effective interval center time, and dimension version semantic fingerprint probability vector with the reference time interval, reference time point, and semantic fingerprint probability vector of the fact candidate record to construct temporal association candidate pairs.

[0054] This implementation scheme achieves precise matching between candidate fact records and candidate dimension versions in the frozen dimension snapshot library by performing consistency correction and time window reconstruction on the timestamps of candidate fact records, locating the effective interval of candidate dimension versions, and combining a three-level pairing method. This effectively constructs a temporal association candidate set and association pairs, improving the accuracy of association pairing. Improving the accuracy and efficiency of association pairing provides a reliable association foundation for subsequent effective association of facts and dimensions, data integration, and data processing, ensuring the consistency and availability of temporal association data.

[0055] Specifically, the process of assessing the degree of temporal mismatch and forming a temporal association dataset is as follows: Add a zero-prevention term to the intersection length, then divide by the union length plus the zero-prevention term to obtain the time interval overlap ratio; take the natural logarithm of the time interval overlap ratio and then take its negative value to obtain the time interval mismatch term; when the overlap ratio approaches 0, the negative logarithm approaches positive infinity; when the overlap ratio approaches 1, the negative logarithm approaches 0; use the Jensen-Shannon divergence as the semantic difference term; the JS divergence itself is dimensionless and ranges from [0,1]. No additional normalization is required; the absolute difference between the reference time base and the center time is calculated, and the absolute difference is divided by the sum of the time offset reference constant and the zero-prevention term, then one is added and the natural logarithm is taken to obtain the time center offset term; dividing by the reference constant eliminates the influence of the absolute time scale, and adding one and taking the logarithm ensures that the term value is close to 0 when the offset is small, and the term value increases logarithmically when the offset is large, maintaining the same dimension as the time interval mismatch term; the time interval mismatch term, semantic difference term, and time center offset term are added together to obtain the temporal join mismatch energy.

[0056] The specific formula for calculating the temporal join mismatch energy is as follows:

[0057] ;

[0058] In the formula, Indicates the first The first fact candidate record and the first Temporal Join mismatch energy between candidate dimension versions is used to characterize the overall degree of mismatch between the current fact candidate record and the current candidate dimension version; Indicates the first The reference time interval for each candidate fact record is used as a time benchmark for overlapping comparison with the effective period of the candidate dimension version; Indicates the first The effective range of each candidate dimension version is used to characterize the effective range of the candidate dimension version on the timeline; The intersection length represents the degree of actual overlap between the two on the time axis. The union length represents the combined span of the two sets over the overall time coverage. Semantic fingerprint probability vector representing candidate fact records semantic fingerprint probability vector with candidate dimension version The Jensen-Shannon divergence between the fact candidate records and the candidate dimension versions is used to characterize the degree of difference in field semantic structure, field distribution pattern and record semantic features between them; Indicates the first The reference time base for each candidate fact record is used to characterize the principal temporal position of the candidate fact record within the reference time interval; Indicates the first The center time of the effective interval of each candidate dimension version is used as the reference time point for offset comparison with the reference time of the fact candidate record; This represents the time offset baseline constant, used to scale the time offset between the current fact candidate record and the candidate dimension version; This indicates the zero-prevention term, obtained by setting a small positive number, with a value range of 10. 8 Up to 10 3 The space between the two is used to avoid cases where the denominator is zero.

[0059] The above three joint assessments have the following technical advantages compared to existing technologies:

[0060] Compared to using only primary key join: Primary key join matches all dimension versions within all time intervals with fact records, resulting in the misassociation of outdated versions with current facts, leading to a large number of incorrect associations; this method suppresses non-overlapping or low-overlapping matching by using time interval mismatch items.

[0061] Compared to using only the overlap ratio of time intervals: relying solely on time overlap can incorrectly associate dimension versions that overlap in time but have completely different field structures or content with fact records, such as associating a new version after a user's address change with an old address fact; this method excludes candidate pairs with semantically mismatched terms by using semantic difference items.

[0062] Compared to using only semantic divergence: relying solely on semantic similarity would associate versions that were semantically similar in the past but were outdated in the actual time with the current facts, such as matching product configuration versions from many years ago with the current facts; this method penalizes candidate pairs with excessive deviations between the reference time and the version center time by using a time center offset term.

[0063] The mismatch energy, which is composed of time interval mismatch term, semantic difference term, and time center offset term, can simultaneously suppress three misassociation scenarios: temporal drift, semantic drift, and interval mismatch, thereby improving the accuracy and robustness of temporal join.

[0064] The temporal join mismatch energy is compared with the mismatch threshold in real time. When the temporal join mismatch energy is less than the mismatch threshold, the current hit version is confirmed as a valid associated version, and the current fact candidate record is formally temporally associated with the valid associated version. During the association, the association timestamp, association confidence, and JS divergence value of the semantic fingerprint probability vectors of both parties are recorded. When the temporal join mismatch energy is greater than or equal to the mismatch threshold, the current fact candidate record is marked as a silent mismatch record and written to the mismatch buffer. The mismatch buffer is organized by object identifier key and event time, and each record saves the mismatch energy value and each sub-item. After completing the above temporal association for all fact candidate records, a temporal association dataset and a silent mismatch tag set are formed, providing input for result release control and replay repair. The temporal association dataset contains associated pair records, association time intervals, and association confidence; the silent mismatch tag set contains mismatch records and corresponding mismatch energy details.

[0065] As shown in Table 1, the fact record temporal matching verification data table is used to quantitatively verify the seven sets of fact records and candidate versions respectively: Fact record R1 corresponds to candidate version V1, with an intersection length of 4, a union length of 6, a time interval mismatch of 0.41, a semantic difference of 0.01, and a temporal join mismatch energy of 0.42; Fact record R2 corresponds to candidate version V2, with an intersection length of 3.8, a union length of 6, a time interval mismatch of 0.46, a semantic difference of 0.06, and a temporal join mismatch energy of 0.55; Fact record R3 corresponds to candidate version V2, with an intersection length of 4.1, a union length of 6.3, a time interval mismatch of 0.43, a semantic difference of 0.05, and a temporal join mismatch energy of 0.52; Fact record R4 corresponds to candidate version V3, The intersection length is 3.6, the union length is 6.2, the time interval mismatch is 0.54, the semantic difference is 0.03, and the tense join mismatch energy is 0.61; Fact record R5 corresponds to candidate version V4, with an intersection length of 4.5, a union length of 6, a time interval mismatch of 0.29, a semantic difference of 0.12, and a tense join mismatch energy of 0.47; Fact record R6 corresponds to candidate version V5, with an intersection length of 4.4, a union length of 6.1, a time interval mismatch of 0.33, a semantic difference of 0.1, and a tense join mismatch energy of 0.49; Fact record R7 corresponds to candidate version V3, with an intersection length of 4, a union length of 6, a time interval mismatch of 0.41, a semantic difference of 0.12, and a tense join mismatch energy of 0.58. Overall data shows that the higher the intersection length and the smaller the mismatch term in the time interval, the lower the overall mismatch energy of temporal join and the better the matching consistency. Among them, R1 has the lowest mismatch energy and the best temporal matching effect, while R4 has the highest mismatch energy and the worst matching consistency.

[0066] Table 1. Fact Record Temporal Matching Verification Data Table

[0067]

[0068] like Figure 4 The temporal join mismatch energy heatmap shown illustrates the distribution of temporal mismatch levels between each candidate fact record and multiple candidate dimension versions. The horizontal axis represents candidate dimension versions, and the vertical axis represents candidate fact records. The values ​​in the colored blocks represent the temporal join mismatch energy calculated between the corresponding candidate fact record and its candidate dimension version. The color scale on the right characterizes the magnitude of the mismatch energy; the higher the value, the greater the mismatch level of the corresponding candidate pair; the lower the value, the better the temporal match. The values ​​marked in red with an asterisk represent the minimum mismatch energy of each candidate fact record across all candidate dimension versions, corresponding to the optimal matching version for that candidate fact record. As shown in the graph, the optimal matching version for R1 is V1, for R2 and R3 it is V2, for R4 and R7 it is V3, for R5 it is V4, and for R6 it is V5. Furthermore, it can be seen that the mismatch energy distribution of different fact candidate records in each candidate dimension version is significantly different, indicating that the present invention does not use a single key value to directly associate, but rather jointly evaluates the degree of overlap of time intervals, the degree of difference of semantic fingerprints and the degree of time center offset, and selects the version with the smallest mismatch energy as the optimal hit result among multiple candidate dimension versions. This enables the identification of silent mismatch problems that are difficult to detect under traditional static association methods, and improves the accuracy and interpretability of temporal association.

[0069] This implementation scheme achieves precise quantification of the temporal mismatch between candidate fact records and candidate dimension versions, effectively suppressing three types of erroneous associations: temporal drift, semantic drift, and interval mismatch. It is significantly superior to traditional methods such as single primary key association, temporal overlap association, or semantic divergence association, greatly improving the accuracy and robustness of temporal join, further ensuring the consistency and reliability of temporal association datasets, and laying a solid foundation for data integration, analysis, and application.

[0070] Specifically, the process of evaluating the release status of the current release cycle based on the temporal correlation dataset is as follows: Based on the temporal correlation dataset, the version consistency rate is obtained by statistically analyzing the proportion of version-consistent results to the total number of results; the number of silent mismatch records is obtained by counting the records that enter the silent mismatch marker set within the current release cycle; the influence radius is obtained by tracking and statistically analyzing the downstream result nodes associated with the silent mismatch records in the current release cycle; the number of all result nodes reachable along the downstream dependency links from the current silent mismatch record is counted once for each node, without duplicate counting; the number of corrected results is obtained by statistically analyzing the number of records that have completed isolation repair, version replay correction, or regenerated candidate release results within the current release cycle; for the same original record, if... The system undergoes multiple operations, including isolation repair, version replay correction, and regeneration, but is counted only once. Different records are counted separately. The previous release gate self-calibration value is exponentially calculated using the gate inertia coefficient to obtain the gate retention term. The sum of the version consistency rate and the zero-prevention term is exponentially calculated by subtracting the gate inertia coefficient from 1 to obtain the consistency contribution term. The number of silent mismatch records is incremented by one and the natural logarithm is taken to obtain the mismatch amplification term. The influence radius is incremented by one and the natural logarithm is taken to obtain the influence amplification term. The mismatch amplification term and the influence amplification term are multiplied together, divided by one, and the number of correction results is added to obtain the mismatch suppression term. The mismatch suppression term is negativeened and exponentially transformed to obtain the risk attenuation term. The gate retention term, consistency contribution term, and risk attenuation term are multiplied sequentially to obtain the release gate self-calibration value. The release gate self-calibration value is used to control whether the results in the temporally correlated dataset enter the formal release layer, preventing the spread of mismatch results to pollute the downstream service layer, and realizing closed-loop control of the reliability and consistency of the data processing system.

[0071] The specific formula for calculating the gate self-correction value is as follows:

[0072] ;

[0073] In the formula, Indicates the first The release gate self-correction value for each release cycle is used to characterize whether the results of the next release cycle have the basis for entering the formal release layer. Indicates the first The release gate self-correction value for each release period is used as the historical state input for the current gate update calculation. The gate inertia coefficient is obtained by statistical analysis of gate fluctuations based on multiple release cycles. Its value ranges from 0.2 to 0.8 and is used to control the retention strength of the gate state in the previous release cycle in the current recursion process. This represents the consistency rate of the current version, which characterizes the degree of consistency among the data versions, rule versions, and dimension versions upon which the results depend in the current release cycle. This indicates the number of silent mismatch records, used to characterize the scale of mismatch issues discovered in the current release cycle; Indicates the radius of influence, used to characterize the spread of the current mismatch result in the downstream propagation chain; This indicates the number of correction results, used to characterize the degree of completion of mismatch repair within the current release cycle; This indicates the zero-prevention term, obtained by setting a small positive number, with a value range of 10. 8 Up to 10 3 The space between the two is used to avoid cases where the denominator is zero.

[0074] like Figure 5 The graph showing the combined evolution of the release period gate value and the radius of influence illustrates the dynamic evolution of the release gate self-correction value as the governance status changes within each release period. The horizontal axis represents the release period, the vertical axis represents the release gate self-correction value, the broken line represents the trajectory of the gate value change in each period, the dashed line represents the release threshold of 0.7, and the bubble size represents the radius of influence of the silent mismatch records in the corresponding period. The number of silent mismatch records (M) and the number of completed repairs (F) are also marked above each bubble. As can be seen from the graph, the release gate self-correction values ​​for periods T1, T3, T5, and T6 are higher than the release threshold, indicating good version consistency and relatively controllable mismatch propagation risk within these periods, providing a strong foundation for formal release. Period T5 has the highest gate value, indicating the optimal overall governance status for this period. Periods T2, T4, and T7 have release gate self-correction values ​​lower than the release threshold, indicating an increase in the number of silent mismatches, an expansion of the impact range, or insufficient repair compensation during these periods, leading to the system automatically tightening release conditions. Period T7 has the lowest gate value, indicating the most significant risk accumulation during this period. Overall, the figure intuitively reflects that the release gate value is not fixed, but dynamically adjusted with changes in the number of mismatches, the spread range, and the degree of repair completion. This demonstrates that the present invention has a closed-loop governance capability of risk perception, dynamic tightening, and recovery control during the result release stage.

[0075] This implementation plan uses multi-dimensional indicators to jointly calculate the release gate self-correction value, enabling dynamic assessment of data quality and risk during the release cycle. It can accurately perceive version consistency, mismatch scale, impact scope, and repair progress, and automatically and dynamically adjust release admission conditions. It effectively avoids the spread of mismatch results downstream, achieving closed-loop reliability control of data release. At the same time, it visually reflects the evolution of the release status, improving the stability, security, and controllability of temporally related data release, and ensuring the consistency and robustness of the overall data governance process.

[0076] Specifically, the process of identifying the release status of the current result is as follows: Based on the release gate self-correction value of the current release period, the release status of the current release period is identified; when the release gate self-correction value is not lower than the release gate threshold, the current result is determined to meet the release conditions; when the release gate self-correction value is lower than the release gate threshold, the current result is determined not to meet the release conditions; when the current result is determined to meet the release conditions, the current set of results to be released is written into the formal release layer; the formal release layer is defined as: a persistent storage area used to store the final results that have passed the release gate verification, providing a read-only access interface to the outside world, and the downstream service layer consumes data from this layer.

[0077] When the current result is determined not to meet the release conditions, the current set of results to be released is written to the isolation buffer, and a freeze flag is set on the affected result nodes, pausing further transmission to the downstream service layer. The isolation buffer is defined as a temporary storage area used to store results that failed the release gate verification and their associated mismatch information. It supports replay correction operations, and the results are not visible to the outside world within this area. The downstream service layer is defined as a collection of business systems that consume data from the formal release layer, including application services, data analysis pipelines, and external APIs, and does not directly access the isolation buffer. At the same time, version replay is performed on the results in the isolation buffer based on the latest frozen dimension snapshot. Using the object identifier key and event time corresponding to the results in the isolation buffer as the basis, the matching dimension version is retrieved again from the latest frozen dimension snapshot library, the temporal association and mismatch energy assessment are re-executed, and candidate release results are regenerated.

[0078] In this implementation plan, by granting access to the intelligent judgment results of the gate self-calibration value, qualified data directly enters the formal release layer to provide stable services to the outside world; unqualified data automatically enters the isolation buffer and is frozen for downstream transmission, effectively preventing the spread of errors. At the same time, based on the latest snapshot, version playback and re-association assessment are automatically performed to achieve closed-loop repair of problematic data, which not only ensures the safety and stability of downstream business, but also improves the overall quality of temporal correlation results and the system's self-healing capability.

[0079] Specifically, the process of generating channel-level response behavior tags is as follows: The current release status, number of silent mismatch records, impact radius, number of correction results, number of replays, and average mismatch energy after replay are structurally bound to generate release governance tags. Release governance tags include: release cycle number, release status, number of mismatches, impact radius, number of repairs, replay flag, and current rule version number. When a continuous release cycle is in a prohibited release state, or the impact radius exceeds the impact radius threshold, a rule review process is triggered to recalibrate the freeze threshold, mismatch threshold, and release gate threshold. The impact radius threshold is a preset constant, defaulting to 10% of the total number of downstream service layer nodes. The specific method for recalibration... The process is as follows: Based on the governance closed-loop record set, the distribution of mismatch energy after passing through the release gate in each period is statistically analyzed. The 25th percentile of the mismatch energy is taken as the new lower limit of the mismatch threshold, and the 75th percentile is taken as the new upper limit of the mismatch threshold. The mismatch threshold for the current period is determined by linear interpolation between the upper and lower limits. The freeze threshold is updated based on the historical quantile of the version confidence coefficient using the same method. The release gate threshold is adaptively adjusted based on the median of the version consistency rate in the historical release period: if the median of the historical consistency rate is higher than 0.9, the threshold is increased by 0.05, and if it is lower than 0.7, the threshold is decreased by 0.05. The release governance tags, isolation records, and replay results generated in the current period are summarized to form the governance closed-loop record set.

[0080] In this implementation plan, governance tags are generated by structurally binding multi-dimensional governance indicators, enabling traceability and quantification of the release status and governance process. When abnormal releases occur or the scope of impact exceeds limits, rule reviews are automatically triggered, and various key thresholds are adaptively corrected, effectively improving the adaptability and accuracy of governance rules. By aggregating and forming a governance closed-loop record set, a closed-loop process from detection, isolation, and replay to rule optimization is achieved, enhancing the system's risk resistance and self-healing capabilities, and ensuring the continuous, stable, and reliable release of temporally related data.

[0081] Specifically, the second aspect of this invention provides a port multi-source business data quality governance system, applied to a port multi-source business data quality governance method, comprising: an acquisition and temporal encapsulation module, used to acquire raw governance data and perform time axis relocation, structure normalization, and encapsulation processing on the raw governance data to obtain a temporal raw dataset; wherein the acquisition and temporal encapsulation module includes a multi-link acquisition unit, a three-timescale alignment unit, a field signature mapping unit, a record role discrimination unit, and a temporal encapsulation unit; and a dimension snapshot freezing module, used to extract candidate dimension records based on the temporal raw dataset, generate a candidate dimension version sequence, evaluate the freezing credibility of the candidate dimension versions, and construct a frozen dimension snapshot library; wherein the dimension snapshot freezing module includes a dimension version sequence generation unit. The system comprises: a meta- and field state vector construction unit, a frozen confidence value calculation unit, and a frozen dimension snapshot storage unit; a temporal correlation verification module, used to perform candidate version recall based on the frozen dimension snapshot library, construct a temporal correlation candidate set, evaluate the degree of temporal mismatch, and form a temporal correlation dataset; the temporal correlation verification module includes a candidate version recall unit, a semantic fingerprint generation unit, a temporal mismatch energy calculation unit, and a correlation pair confirmation unit; and a release closed-loop control module, used to evaluate the release status of the current release cycle based on the temporal correlation dataset, identify the release status of the current result, and generate channel-level response behavior labels; the release closed-loop control module includes a release gate self-correction unit, a release status identification unit, an isolation buffer management unit, and a governance label generation unit.

[0082] This implementation plan achieves unified alignment and structural normalization of multi-source port business data along the timeline, forming a standardized temporal raw dataset. Through a dimensional snapshot freezing mechanism, it automatically generates a reliable sequence of dimensional versions and constructs a high-quality frozen dimensional snapshot library, effectively filtering out unreliable versions. Using temporal correlation verification, it accurately quantifies the degree of temporal mismatch and corrects mismatch relationships, forming a high-confidence temporal correlation dataset. Simultaneously, combined with closed-loop release control, it evaluates the release status in real time and generates channel-level response behavior labels, achieving self-correction of the release gate and anomaly isolation buffering. This end-to-end automated governance significantly reduces manual intervention, lowers data governance costs, and improves the overall quality, temporal consistency, and system robustness of multi-source port business data.

[0083] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0084] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for quality governance of multi-source business data in ports, characterized in that, Includes the following steps: S1. Obtain the raw governance data and perform time axis relocation, structure normalization and encapsulation processing on the raw governance data to obtain the temporal raw dataset; S2, extract candidate records of dimensions based on the original temporal dataset, generate a sequence of candidate dimension versions, evaluate the credibility of freezing the candidate dimension versions, and build a frozen dimension snapshot library; S3, based on the frozen dimension snapshot library, performs candidate version recall, constructs a temporal association candidate set, evaluates the degree of temporal mismatch, and forms a temporal association dataset; The specific process of retrieving candidate versions and constructing a temporally related candidate set based on the frozen dimension snapshot library is as follows: Based on the temporal raw dataset, standard record units with record role identifiers as candidate fact records are extracted. After consistency correction of the event time, collection time, and storage time of candidate fact records within the temporal raw dataset, a reference time interval is reconstructed by combining the time intervals of adjacent candidate fact records with the same object identifier key. The effective interval is obtained by locating the version boundaries of candidate dimension version sequences arranged in event time order under the same dimension object. The intersection length and union length are obtained by calculating the intersection and union of the reference time interval and the effective interval. Semantic field encoding and frequency normalization are performed on candidate fact records and candidate dimension versions respectively: field names and values ​​are extracted from each record to form a word sequence; the frequency of each word is statistically analyzed based on the global bag-of-words model and normalized to a probability distribution, resulting in their respective semantic fingerprint probability vectors. The Jensen-Shannon divergence is obtained by calculating the symmetric divergence based on the intermediate distribution of both. A reference time benchmark is obtained by projecting the event time, collection time, and storage time of candidate fact records after ranking them by credibility. The center time is obtained by finding the median between the start and end times of the effective interval of the candidate dimension version. The absolute offset between the reference time of the candidate fact record in the associated sample and the center time of the hit dimension version is statistically analyzed to obtain the time offset baseline constant; the candidate dimension version set with the same object identifier key is recalled in the frozen dimension snapshot library according to the object identifier key to form the temporal association candidate set; The specific process for assessing the degree of temporal mismatch and forming a temporal association dataset is as follows: Adding the anti-zero term to the intersection length and dividing by the union length plus the anti-zero term yields the time interval overlap ratio. Taking the natural logarithm of the time interval overlap ratio and then taking the negative value yields the time interval mismatch term. The Jensen-Shannon divergence is used as the semantic difference term. The absolute difference between the reference time base and the center time is calculated, and the absolute difference is divided by the sum of the time offset base constant and the anti-zero term, then added and taken as the natural logarithm to obtain the time center offset term. The time interval mismatch term, the semantic difference term, and the time center offset term are added together to obtain the temporal join mismatch energy. Real-time comparison of temporal join mismatch energy and mismatch threshold; If the mismatch energy of temporal join is less than the mismatch threshold, the current hit version is confirmed as a valid associated version, and the current fact candidate record is formally associated with the valid associated version in temporal form; If the temporal join mismatch energy is greater than or equal to the mismatch threshold, the current fact candidate record is marked as a silent mismatch record and written into the mismatch buffer. After completing the above temporal association for all fact candidate records, a temporal association dataset and a silent mismatch mark set are formed, providing input for result release control and replay repair. S4, based on the temporal correlation dataset, evaluates the release status of the current release cycle, identifies the release status of the current result, and generates channel-level response behavior labels; The specific process for evaluating the release status of the current release cycle based on the temporal correlation dataset is as follows: Based on the temporal correlation dataset, the version consistency rate is obtained by statistically analyzing the proportion of the number of results with consistent versions to the total number of results; the number of silent mismatch records is obtained by counting the records that enter the silent mismatch tag set during the current release cycle. The impact radius is obtained by tracking and statistically analyzing the downstream result nodes associated with the silent mismatch records in the current release cycle; the number of correction results is obtained by counting the number of records that have completed isolation repair, version replay correction, or regenerated candidate release results in the current release cycle. The gate retention term is obtained by exponentially calculating the previous gate self-calibration value using the gate inertia coefficient. The consistency contribution term is obtained by exponentially calculating the sum of the version consistency rate and the zero-prevention term by subtracting the gate inertia coefficient from 1. The mismatch amplification term is obtained by adding one to the number of silent mismatch records and taking the natural logarithm. The impact amplification term is obtained by adding one to the radius of influence and taking the natural logarithm. The mismatch suppression term is obtained by multiplying the mismatch amplification term and the impact amplification term, then dividing by one and adding the number of calibration results. The risk attenuation term is obtained by negativening the mismatch suppression term and performing an exponential transformation. The gate retention term, consistency contribution term, and risk attenuation term are multiplied sequentially to obtain the release gate self-calibration value. The specific process for generating channel-level response behavior labels is as follows: The current release status, number of silent mismatch records, radius of influence, number of correction results, number of replays, and average mismatch energy after replay are structurally bound to generate release governance tags; The governance release tags include: release cycle number, release status, number of mismatches, impact radius, number of fixes, replay flag, and current rule version number; When a continuous release cycle is in a prohibited release state, or when the influence radius exceeds the influence radius threshold, the rule review process is triggered to recalibrate the freeze threshold, mismatch threshold, and release gate threshold; the release governance tags, isolation records, and replay results generated in the current cycle are summarized to form a governance closed-loop record set.

2. The method for quality governance of multi-source business data in ports according to claim 1, characterized in that: The specific process of obtaining the original governance data and performing time-axis relocation, structure normalization, and encapsulation on the original governance data to obtain the temporal original dataset is as follows: The database log stream, interface return stream, message queue stream, file import stream, and cache flushing stream in the heterogeneous data link are collected to obtain raw governance data. The raw governance data includes object identification key, source record payload, source side event time, collection time, governance platform storage time, source site identifier, field structure information, and original content summary information. A timeline relocation method based on three-timescale alignment is used to uniformly correct the source-side event time, collection time, and governance platform entry time in the original governance data. A structure normalization method based on field signature mapping is used to uniformly process the field names, field types, null value encoding, and time formats in the original governance data to form standard record units. A candidate labeling method based on record role discrimination is used to classify the standard record units into fact candidate records, dimension candidate records, or correction candidate records. The standard record units after type classification are then temporally encapsulated to generate a temporal raw dataset.

3. The method for port multi-source business data quality governance according to claim 1, characterized in that: The specific process of extracting candidate dimension records based on the temporal raw dataset and generating a candidate dimension version sequence is as follows: Based on the temporal raw dataset, standard record units with record role identifiers as candidate dimension records are selected; candidate dimension records are merged according to object identifier keys to form an object-level candidate version set; candidate dimension records under the same object identifier key are arranged in chronological order of events, and continuous records are versioned to generate a candidate dimension version sequence; adjacency associations are established for adjacent versions in the candidate dimension version sequence, conflict associations are established for candidate dimension versions with overlapping time intervals, and evolutionary associations are established for candidate dimension versions with rollback, replacement, or parallel update relationships to construct a dimension version graph; The duration is obtained by comparing the candidate dimension versions arranged in chronological order under the same dimension object; the freeze time constant is obtained by statistically analyzing the duration of adjacent candidate versions of the same type of dimension object; the field state vector is obtained by concatenating the field value encoding, field non-empty bitmap, field type signature, and field domain distribution parameters corresponding to the candidate dimension version; the state difference is obtained by calculating the L2 norm after subtracting the field state vector of the current candidate version from the field state vector of the previous candidate version; the field state change scale benchmark is obtained by statistically analyzing the state difference between adjacent candidate versions of the same dimension object or the same type of dimension object; the adjacent version set is obtained by extracting the previous and next adjacent candidate dimension versions within the evolution window range under the same dimension object, with the event time of the current candidate version as the center; the interval overlap rate is obtained by calculating the intersection length of the effective interval of the current candidate version and the effective interval of the adjacent version, and then dividing by the sum of the union length of the two and the zero-prevention term.

4. The method for quality governance of multi-source business data in ports according to claim 3, characterized in that: The specific process of evaluating the credibility of freezing candidate dimension versions and constructing a frozen dimension snapshot library is as follows: Divide the duration by the sum of the freeze time constant and the zero-prevention term, then input the result into the hyperbolic tangent function to obtain the duration contribution term; divide the state difference by the sum of the field state change scale benchmark and the zero-prevention term, take the negative, and perform an exponential transformation to obtain the difference suppression term; subtract one from the interval overlap rate between the current candidate version and each adjacent version, and then multiply the results sequentially to obtain the overlap constraint term; multiply the duration contribution term, the difference suppression term, and the overlap constraint term sequentially to obtain the snapshot freeze confidence value; Real-time comparison of snapshot freeze confidence value and snapshot freeze threshold; When the snapshot freeze confidence value is greater than or equal to the snapshot freeze threshold, the current candidate version is written into the frozen dimension snapshot library and marked as the official frozen version. When the snapshot freeze confidence value is lower than the snapshot freeze threshold, the current candidate version is kept in the waiting area for the next round of evaluation before being judged. The selected official frozen versions are indexed and organized by dimension object key, effective start time and effective end time to build a frozen dimension snapshot library.

5. The method for quality governance of multi-source business data in ports according to claim 1, characterized in that: The specific process for identifying the publication status of the current result is as follows: Based on the release gate self-correction value of the current release period, the release status of the current release period is identified; when the release gate self-correction value is not lower than the release gate threshold, the current result is determined to meet the release conditions. When the self-correction value of the release gate is lower than the release gate threshold, the current result is determined not to meet the release conditions. When the current result is determined to meet the publishing conditions, the current set of results to be published is written into the formal publishing layer; When it is determined that the current result does not meet the publishing conditions, the current set of results to be published is written into the isolation buffer, and a freeze flag is set on the affected result nodes to suspend further transmission to the downstream service layer; at the same time, version replay is performed on the results in the isolation buffer based on the latest frozen dimension snapshot to regenerate candidate publishing results.

6. A port multi-source business data quality governance system, employing a port multi-source business data quality governance method as described in any one of claims 1-5, characterized in that, include: The acquisition and temporal encapsulation module is used to acquire raw governance data and perform time axis relocation, structure normalization and encapsulation processing on the raw governance data to obtain the temporal raw dataset; The dimension snapshot freezing module is used to extract candidate dimension records based on the temporal raw dataset, generate a sequence of candidate dimension versions, evaluate the credibility of freezing the candidate dimension versions, and build a frozen dimension snapshot library. The temporal correlation verification module is used to perform candidate version recall based on the frozen dimension snapshot library, construct a temporal correlation candidate set, evaluate the degree of temporal mismatch, and form a temporal correlation dataset; The closed-loop control module is used to evaluate the release status of the current release cycle based on the temporal correlation dataset, identify the release status of the current result, and generate channel-level response behavior labels.

Images