Data association increment determination methods, devices, electronic equipment, media and chips

Abstract

Embodiments of the present invention provide a method, apparatus, electronic device, medium, and chip for determining data association increments. The method includes: acquiring current state fingerprints corresponding to multiple data tables in a target database; determining a state change feature vector corresponding to each data table based on the current state fingerprints and local state data; determining at least one first-type data table and at least one second-type data table; determining a detailed data feature set based on the at least one first-type data table; acquiring a historical key data feature set corresponding to each second-type data table from locally cached data; determining a candidate association set based on the detailed data feature set and the historical key data feature set; and determining the association increment result corresponding to the first-type data table based on the candidate association set. Through the solution of the present invention, accurate perception and incremental calculation of data changes are achieved, thereby significantly improving the efficiency of association discovery.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a method, apparatus, electronic device, medium, and chip for determining data association increments. Background Technology

[0002] In applications that construct a comprehensive data relationship graph, it is often necessary to uncover and complete undefined relationships based on the potential reference relationships between data values ​​in database tables. This typically relies on performing a full scan of all data tables in the database and comparing each pairwise to discover inclusion or matching patterns between column values, thereby inferring possible logical foreign key relationships.

[0003] However, when faced with a large-scale database containing tens of thousands of tables, the computational overhead of the full scan method is extremely large. When the database undergoes minor changes, it is often necessary to re-execute a full database scan and comparison, which cannot accurately detect and locate the scope of the changes, resulting in wasted computing resources and delays in graph updates. This makes it difficult to meet the processing needs for efficient, real-time discovery and maintenance of data relationships. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, electronic device, medium, and chip for determining data association increments, which can solve the problem that when a small number of database changes, a full database scan and comparison need to be re-executed, making it difficult to meet the processing requirements for efficient, real-time discovery and maintenance of data association relationships.

[0005] In view of this, an embodiment of the first aspect of the present invention provides a method for determining data association increments.

[0006] An embodiment of the second aspect of the present invention provides a data association increment determination device.

[0007] An embodiment of the third aspect of the present invention provides an electronic device.

[0008] An embodiment of the fourth aspect of the present invention provides a readable storage medium.

[0009] An embodiment of the fifth aspect of the present invention provides a chip.

[0010] To achieve the above objectives, an embodiment of the first aspect of the present invention provides a method for determining data association increments, comprising: acquiring current state fingerprints corresponding to multiple data tables in a target database, wherein the target database is configured with local cache data and local state data; determining a state change feature vector corresponding to each data table based on the current state fingerprints and local state data; determining at least one first-type data table and at least one second-type data table from the multiple data tables based on the state change feature vectors; determining a detailed data feature set based on at least one first-type data table, wherein the detailed data feature set includes at least the updated state fingerprint and non-primary key column value features; acquiring a historical key data feature set corresponding to each second-type data table from the local cache data, wherein the historical key data feature set includes at least primary key column value features and / or a primary key fingerprint generated based on the primary key column value features; determining a candidate association set based on the detailed data feature set and the historical key data feature set; determining an association increment result corresponding to the first-type data table based on the candidate association set; and updating the local state data based on the association increment result.

[0011] This invention provides a method for determining incremental data associations, aiming to address the high computational cost and low efficiency of full-scale scanning when constructing and updating a global data association graph in a large-scale database environment. Traditional methods typically require a complete scan and pairwise comparison of all tables when detecting potential references between tables. When only a small amount of data in the database has changed, this full-scale processing mode results in significant waste of computational resources and time delays. This invention introduces a state fingerprint comparison and differential processing mechanism to achieve accurate perception and incremental calculation of data changes, thereby significantly improving the efficiency of association discovery.

[0012] In some technical solutions, optionally, the state change feature vector corresponding to each data table is determined based on the current state fingerprint and local state data, including: obtaining the historical state fingerprint corresponding to each data table from the local state data; for each data table, comparing multiple feature values ​​in the current state fingerprint with multiple historical feature values ​​in the historical state fingerprint to determine a first comparison result; and determining a state change feature vector to characterize whether the data table has changed and the dimension of the change based on the first comparison result; wherein, the multiple feature values ​​in the current state fingerprint include row count feature value, primary key maximum value feature value, last update time feature value, and table structure hash feature value.

[0013] Understandably, change detection is upgraded from a simple, misjudgment-prone rule based on a single timestamp or row count to a precise quantitative analysis process based on multi-dimensional feature fusion. This approach significantly improves the accuracy and robustness of change detection, sensitively capturing actual changes in the content of data tables (such as an increase in the number of rows) and reliably identifying changes in structural definitions, while effectively avoiding false triggers caused by minor adjustments to database statistics or cache updates.

[0014] In some technical solutions, optionally, determining at least one first-type data table and at least one second-type data table from multiple data tables based on state change feature vectors includes: comparing each state change feature vector with a preset change judgment threshold to determine a second comparison result; determining data tables whose change amounts corresponding to feature values ​​exceed the corresponding change judgment thresholds as first-type data tables based on the second comparison result; and determining data tables whose change amounts corresponding to feature values ​​do not exceed the corresponding change judgment thresholds as second-type data tables.

[0015] In this scheme, the state change feature vector reflecting multidimensional changes is compared with a set of structured change judgment thresholds dimension by dimension, and the aggregation logic that determines a change if any dimension exceeds the limit is used to accurately divide all data tables into a first type of data table that requires in-depth scanning and a second type of data table that can directly reuse historical features.

[0016] Understandably, the determination of changes in the status of data tables is upgraded from relying on a single, fixed rule to a configurable, multi-dimensional, comprehensive intelligent technology decision-making process, thereby significantly reducing the overall input and output load and computational overhead of the system.

[0017] In some technical solutions, optionally, a detailed data feature set is determined based on at least one first type of data table, including: performing a full table scan or deep sampling scan operation on the first type of data table to obtain full data or sampled data, and extracting data features containing non-primary key column value features from the full data or sampled data to form a detailed data feature set.

[0018] In this scheme, for the first type of data table obtained from the classification, a full table scan with higher resource consumption or a deep sampling scan that balances efficiency and representativeness is selected according to the resource allocation strategy to obtain its latest full or sampled data. From this data, non-primary key column value features that may constitute relationships are extracted and refined to form a detailed set of data features that reflect the latest data content of the table.

[0019] Understandably, by strictly limiting deep scan operations to the first type of data tables that have undergone changes, and introducing deep sampling as an optional optimization method, unnecessary database query load, network transmission overhead, and raw data processing volume are minimized while ensuring that the latest data characteristics of the changed tables can be captured, thereby significantly improving the efficiency of relationship discovery.

[0020] In some technical solutions, optionally, the historical key data feature set corresponding to each second type of data table is obtained from the local cache data, including: performing a lightweight query or directly reading from the cache for the second type of data table to obtain the primary key column value features and form a historical key data feature set.

[0021] In this solution, for the second type of data table obtained from the classification, no full table or deep scan is performed. Instead, the core identification features and primary key column value features that have been pre-calculated and stored are obtained by performing a lightweight database metadata query or directly reading the local cache, and a set of historical key data features for association matching is formed.

[0022] Understandably, transforming the originally necessary and costly database input / output operations into almost overhead memory reads or extremely lightweight metadata queries not only greatly saves computing resources and network bandwidth on the database side, but also significantly reduces the overall response time of association discovery tasks.

[0023] In some technical solutions, optionally, a candidate association set is determined based on a detailed data feature set and a historical key data feature set, including: determining at least one subset of non-primary key column value features based on the detailed data feature set; determining at least one subset of primary key column value features based on the historical key data feature set; performing matching analysis on the non-primary key column value feature subset and the primary key column value feature subset to determine the matching result; determining at least one candidate association based on whether the matching result meets a preset association determination threshold; and determining a candidate association set based on multiple candidate associations.

[0024] In this scheme, a subset of non-primary key column value features is extracted from the detailed features of the changed table, and a subset of primary key column value features is extracted from the cached features of the unchanged table. The matching degree between the two is evaluated through efficient set intersection operation, and whether a valid candidate association relationship is formed is determined based on preset quantity and proportion thresholds. Finally, all associations that meet the conditions are summarized to form a candidate association relationship set.

[0025] Understandably, performing intersection matching by directly manipulating highly condensed feature subsets greatly reduces the amount of data involved in the calculation and improves the speed of association analysis.

[0026] An embodiment of the second aspect of the present invention provides a data association incremental determination device, comprising: a fingerprint acquisition module, configured to acquire current state fingerprints corresponding to multiple data tables in a target database, wherein the target database is configured with local cache data and local state data; a vector determination module, configured to determine a state change feature vector corresponding to each data table based on the current state fingerprint and the local state data; a data classification module, configured to determine at least one first-type data table and at least one second-type data table from the multiple data tables based on the state change feature vector; a full scan module, configured to determine a detailed data feature set based on at least one first-type data table, wherein the detailed data feature set includes at least the updated state fingerprint and non-primary key column value features; a cache loading module, configured to acquire a historical key data feature set corresponding to each second-type data table from the local cache data, wherein the historical key data feature set includes at least primary key column value features and / or a primary key fingerprint generated based on the primary key column value features; an association calculation module, configured to determine a candidate association relationship set based on the detailed data feature set and the historical key data feature set; an incremental determination module, configured to determine the association incremental result corresponding to the first-type data table based on the candidate association relationship set; and a state synchronization module, configured to update the local state data based on the association incremental result.

[0027] An embodiment of the third aspect of this application provides an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the data association increment determination method as described in the first aspect.

[0028] An embodiment of the fourth aspect of this application provides a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the data association increment determination method as described in the first aspect.

[0029] An embodiment of the fifth aspect of this application provides a chip including a processor and a communication interface, the communication interface and the processor being coupled together, the processor being used to run a program or instructions to implement the steps of the data association increment determination method as described in the first aspect.

[0030] Additional aspects and advantages of the technical solutions of the present invention will become apparent in the following description or may be learned by practice of the invention. Attached Figure Description

[0031] Figure 1 One of the flowcharts illustrating the data association increment determination method according to this application is shown;

[0032] Figure 2 A second flowchart illustrating the data association increment determination method according to this application is shown;

[0033] Figure 3 The third flowchart illustrates the data association increment determination method according to this application;

[0034] Figure 4 The fourth flowchart illustrates the data association increment determination method according to this application;

[0035] Figure 5 A schematic block diagram of the data association increment determination device according to this application is shown;

[0036] Figure 6 A schematic block diagram of the structure of an electronic device according to this application is shown;

[0037] Figure 7 A schematic diagram of the algorithm for determining the data association increment based on this application is shown;

[0038] Figure 8 A logical diagram illustrating the memory arrangement for accommodating new data and the old skeleton according to this application is shown.

[0039] Among them, 900: data association incremental determination device; 902: fingerprint acquisition module; 904: vector determination module; 906: data classification module; 908: full scan module; 910: cache loading module; 912: association calculation module; 914: incremental determination module; 916: state synchronization module;

[0040] 1000: Electronic device; 1109: Memory; 1110: Processor. Detailed Implementation

[0041] To better understand the above-described objectives, features, and advantages of the embodiments of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0042] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, embodiments of the invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below.

[0043] When constructing a comprehensive data relationship graph, the cost of full scanning and comparison across thousands of database tables is extremely high. Related foreign key discovery methods typically involve a one-time full calculation. When minor changes occur in the database (such as adding a few rows to a table), the entire database often needs to be rescanned, resulting in significant waste of computational resources and delays in timeliness. There is a lack of an incremental update mechanism that can accurately detect changes and recalculate only the changed parts.

[0044] The data association increment determination method, apparatus, electronic device, medium, and chip provided in this application will be described in detail below with reference to specific embodiments and application scenarios.

[0045] This embodiment provides a method for determining data association increments, such as... Figure 1 As shown, the methods for determining data association increments include:

[0046] Step S100: Obtain the current state fingerprints corresponding to multiple data tables in the target database. The target database has local cache data and local state data set accordingly.

[0047] Step S102: Determine the state change feature vector corresponding to each data table based on the current state fingerprint and local state data;

[0048] Step S104: Determine at least one first-type data table and at least one second-type data table from multiple data tables based on the state change feature vector;

[0049] Step S106: Determine a detailed data feature set based on at least one first-type data table. The detailed data feature set includes at least the updated state fingerprint and non-primary key column value features.

[0050] Step S108: Obtain the historical key data feature set corresponding to each second type of data table from the local cache data. The historical key data feature set includes at least the primary key column value feature and / or the primary key fingerprint generated based on the primary key column value feature.

[0051] Step S110: Determine the candidate association set based on the detailed data feature set and the historical key data feature set;

[0052] Step S112: Determine the incremental association results corresponding to the first type of data table based on the candidate association set;

[0053] Step S114: Update the local status data based on the associated incremental results.

[0054] This invention provides a method for determining incremental data associations, aiming to address the high computational cost and low efficiency of full-scale scanning when constructing and updating a global data association graph in a large-scale database environment. Traditional methods typically require a complete scan and pairwise comparison of all tables when detecting potential references between tables. When only a small amount of data in the database has changed, this full-scale processing mode results in significant waste of computational resources and time delays. This invention introduces a state fingerprint comparison and differential processing mechanism to achieve accurate perception and incremental calculation of data changes, thereby significantly improving the efficiency of association discovery.

[0055] Specifically, the method first obtains the state fingerprints of multiple data tables in the target database.

[0056] A state fingerprint is a lightweight, fast-access set of metadata features used to characterize a snapshot of a data table's state at a given moment.

[0057] Status fingerprints are typically obtained by querying the system table or metadata table in the database.

[0058] For example, the current state fingerprint includes at least the row count feature, the maximum primary key value feature, the last update time feature, and the table structure hash feature.

[0059] State fingerprints are generated from multiple independent metadata features through predefined combination functions, including hash concatenation or feature weighted fusion. By combining multiple features, the sensitivity of state fingerprints to different types of changes in data tables and the uniqueness of their representation are enhanced, reducing the probability of false positives.

[0060] The row count feature reflects the total number of records in the data table; the primary key maximum value feature identifies the latest value of the primary key column; the last update time feature records the most recent modification time of the table data; and the table structure hash feature is obtained by hashing the table structure definition information and is used to detect whether the table structure has changed. These features together constitute a comprehensive quantitative description of the state of a data table.

[0061] In a relational database, a data table is used to store entity information on a specific topic, such as a customer table or an order table.

[0062] Each row in a data table represents an independent entity record, while each column represents a specific attribute of that entity. To efficiently and accurately manage and identify these records, database design introduces the concept of keys, the most basic and important of which are primary keys and non-primary keys.

[0063] A primary key is a column or a set of columns used to uniquely and definitively identify each row of records in a data table. A primary key must satisfy the following core characteristics: first, uniqueness, meaning that no two rows in the entire data table can have the same primary key value; second, non-nullability, meaning that each value of the primary key cannot be null; and third, stability, ideally, once a primary key value is assigned, it should not be changed.

[0064] When another table needs to reference records in this table, a connection is established by storing the primary key value of this table; this referencing column is called a foreign key.

[0065] Non-primary key refers to all columns in a data table other than the primary key column.

[0066] These non-primary key columns are used to describe various attributes and characteristics of an entity, and their functions and constraints are far less stringent than those of primary keys. Non-primary key columns can be broadly categorized as follows: first, foreign key columns, which store the primary key value of another table and are used to establish relationships between tables; second, ordinary data columns, which directly store business data, such as "name," "age," and "major" in the "student table"; and third, other constraint columns that may exist, such as unique constraint columns. The values ​​of non-primary key columns can be non-unique, can be null, and can be updated according to business needs.

[0067] In summary, the primary key is the core identifier of a table, responsible for ensuring the uniqueness and addressability of rows and supporting the relationship network between tables; while non-primary keys are the rich information carriers of a table, responsible for storing the specific content that constitutes the business entities. The two complement each other, jointly forming the structure and semantics of a data table. This invention detects the addition and deletion of records by monitoring changes in the primary key column, and discovers potential new data relationships by analyzing changes in the characteristics of non-primary key column values.

[0068] While acquiring the current state fingerprint, the system accesses pre-established local state data.

[0069] Local state data is persistently stored on the local disk or in a cache. Its core content consists of the historical state fingerprints of all data tables saved at the end of the last processing cycle. By comparing the currently acquired state fingerprint with the corresponding historical state fingerprints in the local state data one by one, the state change feature vector for each data table is determined.

[0070] This determination process includes, for each data table, performing mathematical difference calculations or logical consistency comparisons between the various feature values ​​in its state fingerprint and the corresponding historical feature values ​​in its historical state fingerprint. This includes, but is not limited to: calculating the difference between the current row number and the historical row number, determining whether the current maximum value of the primary key is greater than the historical value, and checking whether the table structure hash values ​​are equal.

[0071] Based on these comparison results, the system generates a state change feature vector, which quantitatively expresses the changes in the data table in various dimensions, such as whether the number of rows has changed, whether the primary key range has expanded, and whether the structure has changed, so as to accurately characterize the degree of change of the data table in subsequent steps.

[0072] Based on the aforementioned generated state change feature vector, two different sets of data tables are determined from multiple data tables, namely at least one first-type data table and at least one second-type data table.

[0073] This determination process is based on a preset change judgment threshold.

[0074] The system compares the changes in each dimension of the state change feature vector of each data table with pre-defined change judgment thresholds for the corresponding dimensions. Based on the comparison results, data tables whose changes in any feature dimension exceed the corresponding threshold are classified as Type I data tables. These data tables are considered to have undergone substantial changes.

[0075] Conversely, data tables whose changes across all feature dimensions do not exceed the corresponding thresholds are classified as Type II data tables. These tables are considered to have remained unchanged or undergone negligible changes since the last processing.

[0076] For example, the first type of data table includes, but is not limited to, dirty tables, and the second type of data table includes, but is not limited to, clean tables.

[0077] For the first type of data table that has been identified, the system will perform a resource-intensive operation to determine its detailed set of data characteristics.

[0078] Specifically, the system initiates a full table scan or deep sampling scan operation for each Type I data table. A full table scan reads all data rows in the table, while deep sampling reads a sufficiently representative portion of the data. From these acquired full or sampled data, the system extracts data features containing non-primary key column value characteristics. These data features may include the set of all unique values ​​appearing in a column, a histogram of value distribution, or specifically coded fingerprint information. All these latest data features extracted from the changed table together constitute the table's detailed data feature set. This set naturally also includes the latest state fingerprint of the Type I data table, i.e., the updated state fingerprint.

[0079] On the other hand, for the identified second-type data tables, the system employs a low-resource-consumption operation to obtain their characteristic information: direct reading from the local cache. The local cache stores the key characteristics of these unchanged tables calculated and stored during the last processing. The system performs lightweight queries or reads directly from memory or disk caches on the second-type data tables to obtain their historical key data feature set. The core of this set is at least one primary key fingerprint, which typically refers to all values ​​or hash representations of the primary key column of the table. Since these tables are determined to be unchanged, their primary key sets are highly likely to be stable, thus allowing direct reuse of historical calculation results and avoiding expensive database input / output operations.

[0080] After obtaining the detailed data feature set of the changed table and the historical key data feature set of the unchanged table, the system begins to perform incremental calculations of the relationships.

[0081] First, extract at least one subset of non-primary key column value features from the detailed data feature set of the first type of data table. These subsets represent the value features that may serve as candidate foreign key columns. Simultaneously, directly obtain at least one subset of primary key column value features from the historical key data feature sets of all second type of data tables.

[0082] The extracted subset of non-primary key column values ​​is intersected with the subset of primary key column values. The purpose of this intersection operation is to discover whether there are any sets of values ​​for non-primary key columns in the new data of the modified table that highly overlap with the sets of values ​​for the primary keys of a table that has not been modified.

[0083] Based on the matching results obtained from the intersection operation, the system applies preset association determination thresholds for judgment. Association determination thresholds typically include quantity thresholds and proportion thresholds.

[0084] For example, the system requires that the number of matched records is greater than a minimum value, and that the matched values ​​account for more than 60% of the candidate non-primary key column values. If the matching results meet these threshold conditions, the system determines that there is a candidate association between the non-primary key column of the first type of data table and the primary key column of the second type of data table. Summarizing all such relationships that meet the conditions forms the set of candidate associations discovered in this incremental calculation.

[0085] Based on this newly discovered set of candidate relationships, the final incremental relationship results corresponding to the first type of data table are determined. This determination process typically involves comparing the set of candidate relationships with historical relationship records, filtering out entirely new relationships that did not exist previously. These newly added relationships are the output of this incremental update, i.e., the incremental relationship results. This result accurately reflects the new inter-table reference relationships that have arisen since the last processing due to data changes.

[0086] The local state data is updated based on the results of this incremental calculation to complete a processing loop. The update includes two aspects.

[0087] First, the latest state fingerprints (i.e., the updated state fingerprints) of all first-type data tables are written to the local state data, overwriting the old historical state fingerprints. This way, these tables will be used as a benchmark for comparison in the next processing cycle.

[0088] Secondly, during the association calculation process, if it is found that a second-type data table, although its own data has not changed, has its primary key referenced by a new first-type data table, then the second-type data table may need to update its feature information in the local cache due to this referencing relationship. The system will identify these affected second-type data tables based on the incremental association results and trigger pull and feature update operations to ensure that the local cached data is consistent with the latest association graph state.

[0089] Through the above process, this invention achieves a hybrid computing mode that performs deep scanning only on changed data and reuses historical features for unchanged data. While ensuring the accuracy of relational discovery, it significantly reduces database input / output load and overall computational overhead, enabling efficient and real-time maintenance of the global data relational graph for large-scale databases.

[0090] In some embodiments, the local cached data optionally employs a tiered storage strategy, storing the historical key data feature set of the most frequently used second-type data tables in memory, while storing less frequently used data on the local disk. Simultaneously, cached items in memory that have not been accessed for an extended period are evicted. This ensures fast read speeds while effectively controlling memory usage and preventing system resource exhaustion.

[0091] In some embodiments, optionally, the association determination threshold used when determining candidate associations based on matching results is dynamic, depending on the data type and cardinality of the non-primary key columns involved in the comparison, as well as the primary key cardinality of the second type of data table. By introducing a dynamic threshold mechanism, association discovery can adapt to the data distribution characteristics of different columns, improving the accuracy of relationship determination.

[0092] In some embodiments, the incremental association results may include not only the newly added association pairs, but also confidence scores, supporting data examples, and the change operation type of the first type of data table associated with the association. By outputting rich incremental contextual information, downstream data graph construction or quality inspection systems can process and use these new associations more intelligently.

[0093] In some embodiments, optionally, such as Figure 2 As shown, step S102: Determine the state change feature vector corresponding to each data table based on the current state fingerprint and local state data, including:

[0094] Step S1020: Obtain the historical state fingerprint corresponding to each data table from the local state data;

[0095] Step S1022: For each data table, compare multiple feature values ​​in the current state fingerprint with multiple historical feature values ​​in the historical state fingerprint to determine the first comparison result;

[0096] Step S1024: Based on the first comparison result, determine the state change feature vector used to characterize whether the data table has been changed and the changed dimensions;

[0097] The current state fingerprint contains several features, including the number of rows, the maximum value of the primary key, the last update time, and the table structure hash.

[0098] In this embodiment, by obtaining the historical state fingerprint of each data table from local persistent storage, the currently collected row count, primary key maximum value, last update time, and table structure hash are precisely compared with historical values ​​using difference calculation and consistency comparison. Based on this, a structured state change feature vector is generated. This vector comprehensively and quantitatively represents the specific changes of each data table in multiple independent dimensions such as record count, primary key range, data freshness, and structural definition.

[0099] Understandably, change detection is upgraded from a simple, misjudgment-prone rule based on a single timestamp or row count to a precise quantitative analysis process based on multi-dimensional feature fusion. This approach significantly improves the accuracy and robustness of change detection, sensitively capturing actual changes in the content of data tables (such as an increase in the number of rows) and reliably identifying changes in structural definitions, while effectively avoiding false triggers caused by minor adjustments to database statistics or cache updates.

[0100] Specifically, the status fingerprint includes, but is not limited to, row number feature value, primary key maximum value feature value, last update time, and table structure hash feature value.

[0101] Multiple feature values ​​can also include the total storage space usage of the data table. When calculating the difference, the current storage space usage is compared with the historical storage space usage to capture physical storage changes caused by data updates, row width changes, or defragmentation, so that the state change feature vector can more comprehensively reflect the physical state changes of the data table.

[0102] The determination of the state change feature vector begins with obtaining the historical state fingerprint corresponding to each data table from the local state data.

[0103] Local state data is a persistent storage structure, usually existing in the form of files or database tables. Its core is to record the state information saved after taking a snapshot of all data tables at a certain reference point in the past (usually when the last full or incremental processing ended successfully).

[0104] Historical state fingerprints are sets of historical features read from records that correspond one-to-one with each data table currently being processed.

[0105] After obtaining the current state fingerprint and the corresponding historical state fingerprint, a core comparative analysis is performed for each data table. This process is not a simple Boolean judgment, but a detailed and quantitative comparison of multiple key feature values ​​that constitute the state fingerprint. This comparison is specifically manifested in two technical operations: difference calculation and consistency comparison. For numerical features with ordinal relationships, the system performs difference calculation. For example, for the row count feature value, the calculation method is to subtract the historical row count from the current row count, resulting in an integer difference that can be positive or negative. This difference directly quantifies the increase or decrease in the number of records. For the primary key maximum value feature value, difference calculation is usually also performed, but a more common judgment is to confirm whether the current value is greater than the historical value. This is essentially a judgment of whether the difference is greater than zero, used to identify whether the primary key range has undergone positive expansion. For the last update time feature value, the comparison is usually based on the order of timestamps to determine whether the current timestamp is later than the historical timestamp, which can be transformed into the calculation of time difference.

[0106] The result of the difference calculation undergoes a smoothing or normalization process before being used to generate the state change feature vector. For example, the absolute difference in the number of rows is divided by the previous historical row count to convert it into a relative rate of change; or the change in storage space is normalized according to the table size. This makes the degree of change between data tables of different sizes comparable, laying the foundation for setting a unified, size-independent threshold for determining changes.

[0107] For unordered hash-type features that represent structural information, a consistency comparison is performed. The table structure hash feature value is a fixed-length string generated by standardizing the data table's schema definition and then applying a hash function. The consistency comparison checks whether the currently calculated table structure hash value is exactly equal to the historically stored table structure hash value. If they are equal, it indicates that the table structure has not changed; if they are not equal, it indicates that the table structure has undergone definitional changes such as adding or deleting columns or modifying column types.

[0108] For example, suppose a data table named "Order Table" has a historical state fingerprint with a row count feature value of 10000, a primary key maximum value feature value of 10000, and a table structure hash feature value of a1b2c3d4.

[0109] In the current scan, the obtained status fingerprint shows a row count of 10050, a primary key maximum value of 10050, a table structure hash of a1b2c3d4, and a last update time feature value changed from 2023-10-01 10:00:00 to 2023-10-05 14:30:00. Therefore, the comparison results for the table will be: a row count difference of +50, a primary key maximum value difference of +50 (or judged as "increased"), consistent table structure hashes, and the last update time has been updated.

[0110] Based on the first comparison result of the difference calculation or consistency comparison for each feature value, these scattered comparison results are integrated into a standardized judgment criterion that can be processed by a machine. This is the process of determining the feature vector of state change.

[0111] A state change feature vector is a structured data object where each dimension corresponds to a feature of the state fingerprint, and its value is a quantized or encoded representation of the feature comparison result. For example, the row count feature dimension can store the calculated difference, the primary key maximum value dimension can store an identifier such as "increased", "unchanged", or a specific increment value, the table structure hash dimension can store a boolean value "true" or "false" indicating whether they are consistent, and the last update time dimension can store a time difference or an "updated" identifier.

[0112] The core function of the generated state change feature vector is to accurately characterize whether the corresponding data table has changed and, if so, what dimensions of change have occurred in subsequent steps. The state change feature vector provides unambiguous input for automated decision-making logic, enabling the system to determine, based on objective data differences rather than pre-defined simple rules, whether a table should be included in the set of changed tables requiring rescanning or in the set of unchanged tables where historical features can be reused. Through this refined feature vector generation mechanism, this invention lays a reliable and efficient data foundation for the entire incremental update process.

[0113] In some embodiments, the consistency comparison optionally includes segmented comparison of the hash feature values ​​of the table structure. Specifically, the table structure definition information is split into multiple sub-parts such as column definitions, index definitions, and constraint definitions, and sub-hash values ​​are calculated for each. During the comparison, not only is the consistency of the overall hash value checked, but when the overall value is inconsistent, the hash values ​​of each sub-part are further compared, thereby accurately identifying whether the structural change is caused by the addition or deletion of columns or changes to indexes, so that the generated state change feature vector can carry more specific change dimension information.

[0114] In some embodiments, optionally, such as Figure 3 As shown, step S104: determining at least one first-type data table and at least one second-type data table from multiple data tables based on the state change feature vector, including:

[0115] Step S1040: Compare the feature vector of each state change with the preset change judgment threshold to determine the second comparison result;

[0116] Step S1042: Based on the second comparison result, the data table whose change in the feature value exceeds the corresponding change judgment threshold is identified as the first type of data table;

[0117] Step S1044: The data table whose change in the corresponding feature value does not exceed the corresponding change judgment threshold is identified as the second type of data table.

[0118] In this scheme, the state change feature vector reflecting multidimensional changes is compared with a set of structured change judgment thresholds dimension by dimension, and the aggregation logic that determines a change if any dimension exceeds the limit is used to accurately divide all data tables into a first type of data table that requires in-depth scanning and a second type of data table that can directly reuse historical features.

[0119] Understandably, this upgrades the determination of data table change status from relying on a single, fixed rule to a configurable, multi-dimensional, comprehensive intelligent technical decision-making process. By pre-setting refined thresholds for different feature dimensions, the system can flexibly adapt to the technical characteristics and sensitivity requirements of different business tables. This ensures high sensitivity in change detection, effectively capturing substantial changes in any dimension, while avoiding misjudgments and omissions through clear rules. This automatic and accurate classification mechanism is a prerequisite for the efficient allocation of computing resources in the entire incremental update process, ensuring that deep scanning resources are precisely deployed only to data tables that have changed, thereby significantly reducing the overall system's input / output load and computational overhead.

[0120] Specifically, the state change feature vector generated in the previous step for each data table is compared with the preset change judgment threshold.

[0121] The state change feature vector is a structured data object, with each dimension corresponding to a change feature, such as the change in the number of rows, the increment of the primary key maximum value, and the Boolean value for table structure consistency.

[0122] The change judgment threshold is also a structured set of preset parameters, in which a critical value for judging whether a "significant change" has occurred is predefined for each dimension of the feature vector.

[0123] The setting of this threshold set is based on technical considerations. For example, the threshold for changes in the number of rows may be set as an absolute value such as 1,000 rows, or as a relative value such as 5%, depending on the system's technical definition of "significant data growth" and the requirements for processing sensitivity; the threshold for changes in table structure is usually "inconsistency" that triggers it.

[0124] The essence of alignment is a vectorized comparison operation. The system compares the actual value of the feature vector in each dimension with the preset threshold of the dimension. This operation may be a comparison operation of greater than, less than or equal to.

[0125] Based on the technical comparison operation, the system will generate a deterministic second comparison result for each data table. The second comparison result is not a simple yes or no, but a set of Boolean judgments corresponding to each feature dimension, used to identify whether the actual amount of change in each dimension exceeds its corresponding technical tolerance range.

[0126] For example, the second comparison result might show: the row number change feature dimension "exceeds the threshold", the primary key change dimension "does not exceed the threshold", and the structural consistency dimension "exceeds the threshold".

[0127] The final technical type of each data table is determined based on the second comparison result: if any feature dimension in the state change feature vector of a data table has a change exceeding the preset change judgment threshold for that dimension, then the data table is identified as a first-type data table. This means that a first-type data table has undergone a change exceeding the preset tolerance in at least one monitored technical dimension. Based on this, the system technically infers that its data content or structure has likely undergone substantial changes, and therefore needs to be marked for subsequent resource-intensive processing, i.e., full or deep scanning.

[0128] Conversely, if the changes in all feature dimensions of a data table's state change feature vector are determined to be within the corresponding change judgment threshold after being compared with their respective thresholds, then the data table is identified as a second type of data table.

[0129] The second type of data table remains stable or experiences only minor fluctuations with no significant technical impact across all monitored key dimensions. Based on this, the system infers that it has been in a clean state since the last snapshot, and its historical data characteristics remain valid. Therefore, in subsequent processing, expensive data reads should be skipped, and locally cached feature information should be reused instead.

[0130] For example, suppose the change determination threshold is set as follows: the number of rows changes by more than 50, the maximum value of the primary key must not have increased, and the table structure hash must be consistent. For a data table whose state change feature vector shows a row count difference of +30, no increase in the primary key, and inconsistent structure hash, its comparison result is: row count change "not exceeding the threshold", primary key change "not exceeding the threshold", and structure change "exceeds the threshold". Based on the logic that any dimension exceeding the threshold is considered a change, this table is identified as a first-type data table due to its inconsistent structure hash. Another data table whose feature vector shows a row count difference of +5, no increase in the primary key, and consistent structure hash, has all dimensions compared and found to be within the threshold, therefore it is identified as a second-type data table.

[0131] In some embodiments, the change determination threshold is not globally uniform, but is associated with the historical change patterns or data size of each data table.

[0132] The system maintains adaptive threshold parameters for different data tables. For tables with frequent historical changes, the threshold is dynamically increased to avoid frequent deep scans triggered by minor fluctuations. For tables with large data volumes, the threshold for row number changes may be a relative proportion rather than an absolute value. Through this adaptive mechanism, while maintaining sensitivity to change detection, misclassification caused by normal minor data fluctuations is significantly reduced, improving the overall processing efficiency of the system.

[0133] In some embodiments, optionally, the judgment logic for a feature value change exceeding the corresponding change judgment threshold supports combined AND and OR conditions. The system allows users or administrators to configure more complex judgment rules for specific data tables. For example, it can be defined that a table is classified as a first-type data table only when "the number of rows changes exceeds the threshold and the table structure changes," or it can be defined that a table is classified only when "the maximum value of the primary key changes exceeds the threshold or the last update time changes exceeds the threshold." This flexible rule configuration capability allows the classification strategy to more accurately match the data characteristics and governance requirements of different business tables.

[0134] In some embodiments, optionally, determining a detailed data feature set based on at least one first type of data table includes: performing a full table scan or deep sampling scan operation on the first type of data table to obtain full data or sampled data, and extracting data features containing non-primary key column value features from the full data or sampled data to form a detailed data feature set.

[0135] In this embodiment, for the first type of data table obtained by classification, a full table scan with higher resource consumption or a deep sampling scan that balances efficiency and representativeness is selected to be performed according to the resource allocation strategy, so as to obtain its latest full or sampled data, and refine and extract the non-primary key column value features that may constitute the relationship, and finally form a detailed data feature set that reflects the latest data content of the table.

[0136] Understandably, by strictly limiting deep scan operations to the first type of data tables that have undergone changes, and introducing deep sampling as an optional optimization method, unnecessary database query load, network transmission overhead, and raw data processing volume are minimized while ensuring that the latest data characteristics of the changed tables can be captured. This enables accurate perception and incremental calculation of data changes, thereby significantly improving the efficiency of relationship discovery.

[0137] The first type of data table is a collection of data tables that have undergone significant changes.

[0138] A full table scan refers to sending a query command to the database to retrieve the complete data content of every row and column in the target table. This operation ensures that 100% of the data characteristics are obtained, but it consumes a large amount of database input / output resources and network bandwidth.

[0139] Deep sampling scanning is an optimized technique that uses specific sampling algorithms, such as primary key-based random sampling, systematic interval sampling, or data page-based sampling, to select a sufficiently representative subset of data rows from the target table, rather than all of them.

[0140] The sampling rate may be determined based on the table size, historical data distribution characteristics, or the system's preset resource allocation strategy. The purpose of choosing deep sampling is to significantly reduce resource consumption during the data acquisition phase, while still being able to infer the overall data characteristics of the table with high confidence.

[0141] By performing a scan, full or sampled data from the first type of data table was obtained from the target database. This data is a raw collection of records containing actual business values, serving as the foundational material for subsequent feature calculations.

[0142] Next, we need to extract the core information for relationship discovery from the full or sampled data we have acquired, namely, data features that include non-primary key column values.

[0143] Non-primary key columns refer to all columns in a data table other than the primary key columns.

[0144] Extracting column value features refers to analyzing the distribution, uniqueness, and patterns of data values ​​in one or more non-primary key columns of interest using algorithms, and then transforming this information into a compact and computable representation.

[0145] Common feature extraction methods include, but are not limited to: calculating the set of all unique values ​​in a column; generating a statistical histogram of column data values; or using algorithms such as Bloom filters and minimum hashing to generate a "fingerprint" or "signature" of the column value set.

[0146] These extracted features collectively constitute an understanding of the current data content of the table, especially the content that may serve as candidate foreign key columns.

[0147] The column value features extracted from all the non-primary key columns of interest in a first-type data table are organized and encapsulated with the latest state fingerprint of the data table (which can be calculated after obtaining full or sampled data) to form a detailed data feature set for the data table. This detailed data feature set is a structured data object containing a condensed feature representation of the complete data snapshot of the changed table at the latest point in time. It serves as the direct input for subsequent mixed association calculations with historical features of the unchanged table. Through this step, the invention transforms the massive amount of original table data into a lightweight yet information-intensive feature representation, preparing the data for subsequent efficient association calculations.

[0148] In some embodiments, the depth sampling scan operation may optionally employ an adaptive sampling strategy, dynamically calculating the sampling rate required for this scan based on the data size of the first type of data table, the frequency of historical changes, and the importance score of the table in the association graph.

[0149] For tables that are large in size but have undergone minor changes, a lower sampling rate is used; for tables with critical business operations or those that have changed structure, a higher sampling rate is used or a full table scan is triggered directly. Through this adaptive resource allocation, the resource consumption of the scanning operation itself is further optimized while ensuring that the extracted data features have sufficient statistical representativeness.

[0150] In some embodiments, optionally, before performing the scan operation, the system determines the scope of data feature extraction based on whether the hash feature value of the table structure of the first type of data table has changed. If the table structure remains unchanged, only column value features related to the data row content are extracted; if the table structure has changed, for example, a new column has been added, the definition information of the new column is recorded while extracting the column value features, and it is included as part of the metadata features in the detailed data feature set. This allows the output feature set to simultaneously reflect changes in data content and structural patterns, providing information for more comprehensive correlation analysis.

[0151] In some embodiments, the scanning and feature extraction process of the first type of data tables can optionally be executed in parallel. The system distributes multiple first type data tables to different processing threads or computing nodes, and simultaneously initiates scanning queries and feature calculations. After each parallel task completes feature extraction, it immediately performs a memory degradation operation, that is, it retains only core results such as non-primary key column value features in memory and releases the original, large-volume full or sampled data. This mechanism improves overall throughput by utilizing parallel computing while strictly controlling the memory usage of individual tasks, preventing memory overflow during peak processing.

[0152] In some embodiments, optionally, obtaining a set of historical key data features corresponding to each second type of data table from local cache data includes: performing a lightweight query or directly reading from the cache on the second type of data table to obtain primary key column value features, forming a set of historical key data features.

[0153] In this embodiment, for the second type of data table obtained by classification, no full table or deep scan is performed. Instead, the core identification features and primary key column value features that are pre-calculated and stored are obtained by performing a lightweight database metadata query or directly reading the local cache, and a set of historical key data features for association matching is formed.

[0154] Understandably, transforming the originally necessary and costly database input / output operations into almost overhead memory reads or extremely lightweight metadata queries not only greatly saves computing resources and network bandwidth on the database side, but also significantly reduces the overall response time of association discovery tasks.

[0155] Specifically, the objects of lightweight queries or direct reads from the cache are the second type of data tables, which are the sets of data tables that have been determined to have not undergone significant changes across all monitoring dimensions after comparing the state change feature vector with the change judgment threshold.

[0156] Since the data content of these tables is presumed to have remained stable since the last processing, there is no need to repeatedly perform expensive data scans; instead, historical feature information that has already been calculated and saved can be reused.

[0157] The system retrieves a set of historical key data features corresponding to each second-type data table from the local cache data. The local cache data here is a storage area located in the application server's local memory or high-speed disk, and its content consists of snapshots of data features pre-calculated and persistently saved for all data tables in past processing cycles.

[0158] The acquisition operation is specifically manifested in two efficient technical means.

[0159] One approach is to execute lightweight queries, which involve sending a specially designed query to the database that targets only specific columns of the target table, typically primary key columns, and usually has row count limitations or only queries metadata. For example, a query might only select primary key columns or only ask for the maximum value of the primary key. These types of queries can be responded to by the database very quickly, generating almost no input / output load.

[0160] The second approach is to read directly from the cache, meaning without interacting with the target database at all, but instead retrieving previously stored feature objects directly from the local cached data structure using key-value pairs such as table names. The core objective of both methods is the same: to obtain the data necessary for association matching with minimal resource overhead.

[0161] Through the above operations, the primary key column value features of the historical key data feature set are obtained.

[0162] The primary key column is the column that uniquely identifies each row of a record in a table.

[0163] Primary key column value characteristics are a machine-processable abstract representation of all possible values ​​in a column. Their purpose is to quickly determine whether a value exists in the primary key set during subsequent set operations.

[0164] The primary key column values ​​obtained from each second-type data table will be organized into a structured set of historical key data features.

[0165] Historical key data feature sets are pre-prepared sets for joins and comparisons on unchanged tables. Unlike detailed data feature sets extracted from Type 1 data tables, which contain various non-primary key column value features, historical key data feature sets are typically more concise.

[0166] That is, the acquisition of historical key data feature sets skips data reading and directly loads only the key features used for association matching, such as the primary key set, from the local cache or history, without performing database input / output operations.

[0167] This differentiated strategy allows the system's overall computing resources to be precisely focused on the changed data, thereby achieving an order-of-magnitude improvement in overall correlation discovery efficiency.

[0168] In some embodiments, the locally cached data optionally employs a tiered storage architecture. Historical key data feature sets from frequently accessed second-type data tables reside in the application server's memory to ensure millisecond-level read speeds; feature sets with lower access frequencies are stored on the local solid-state drive. The system dynamically adjusts the data distribution between memory and disk based on recent access patterns. This tiered strategy effectively controls overall memory resource usage while ensuring high performance for the vast majority of reads.

[0169] In some embodiments, the primary key column value features may be represented and stored using multiple complementary data structures. For example, a complete ordered list of primary key values ​​may be stored simultaneously for exact matching, and the corresponding Bloom filter may be stored for fast negation. After retrieving features from the cache, the system selects the most suitable data structure for use based on the specific scenario of subsequent association calculations. By maintaining multiple representations, an optimized foundation is provided for matching operations with different precision and performance requirements.

[0170] In some embodiments, locally cached data can optionally be preloaded during system startup or task scheduling. Based on historical task records or a predefined configuration list, the system preloads a set of historical key data features from a batch of high-frequency or critical second-type data tables from disk into memory. This enables instantaneous memory reads for feature acquisition operations on these tables after the formal processing begins, further reducing the overall latency of the task.

[0171] In some embodiments, optionally, such as Figure 4 As shown, step S110: Determine the candidate association set based on the detailed data feature set and the historical key data feature set, including:

[0172] Step S1102: Determine at least one subset of non-primary key column value features based on the detailed data feature set;

[0173] Step S1104: Determine at least one subset of primary key column value features based on the historical key data feature set;

[0174] Step S1106: Perform a matching analysis between the non-primary key column value feature subset and the primary key column value feature subset to determine the matching results;

[0175] Step S1108: Determine at least one candidate association relationship based on whether the matching result meets the preset association determination threshold;

[0176] Step S1110: Determine the set of candidate associations based on multiple candidate associations.

[0177] In this embodiment, a subset of non-primary key column value features is extracted from the detailed features of the changed table, and a subset of primary key column value features is extracted from the cached features of the unchanged table. The matching degree between the two is evaluated through efficient set intersection operation, and whether a valid candidate association relationship is formed is determined based on preset quantity and proportion thresholds. Finally, all associations that meet the conditions are summarized to form a candidate association relationship set.

[0178] Understandably, directly manipulating highly condensed feature subsets for intersection matching significantly reduces the amount of data involved in the computation, thus improving the speed of association analysis. Simultaneously, the introduction of a configurable dual-threshold determination mechanism for both quantity and proportion provides a statistically significant confidence guarantee for the establishment of associations, effectively filtering out accidental matches and noise, and ensuring the accuracy and reliability of the discovered associations.

[0179] First, determine at least one subset of non-primary key column value features based on the detailed data feature set.

[0180] The detailed data feature set is obtained by full or deep sampling scanning and extraction from the first type of data table that has been determined to have changed, and includes the value features of each non-primary key column of the data table.

[0181] The process of determining a subset of non-primary key column values ​​is a strategy-based filtering action.

[0182] The system may select all non-primary key column features in the set, or it may filter the columns most likely to be foreign key candidates based on the column's data type, name pattern, or historical association probability, such as columns named "xxx_id", "xxx_code", or those storing numeric identifiers. Each determined subset represents a certain characteristic representation of the complete set of values ​​from a specific non-primary key column in a change table.

[0183] At the same time, at least one subset of primary key column value features is determined based on the set of historical key data features.

[0184] The historical key data feature set is obtained from the local cache of the Type II tables that have been determined to be unchanged, and its core is the primary key fingerprint. Determining a subset of primary key column value features usually means directly using this primary key fingerprint, since each Type II table typically has only one primary key. Therefore, each such subset represents the characteristics of a set of unique identifiers for an unchanged table.

[0185] Next, the system performs the core association discovery calculation, which involves matching the non-primary key column value feature subset with the primary key column value feature subset. The matching analysis includes, but is not limited to, intersection operations.

[0186] The intersection operation operates on a subset of features, not the original data values. The purpose of the operation is to quickly estimate the degree of overlap between the set of values ​​of a non-primary key column in a changed table and the set of values ​​of the primary key in an unchanged table.

[0187] Depending on the different data structures used for the feature subsets, the specific implementation of the intersection operation varies.

[0188] For example, if the feature subset is a Bloom filter, the operation checks whether each value (or its hash) of a non-primary key column in the change table can be found in the Bloom filter representing the primary key set, and counts the number of "likely hits". If the feature subset is a min-hash signature, the operation calculates an estimate of the similarity between two signatures. The result of this operation is quantified into a matching result, which typically includes a match count and a match percentage.

[0189] After obtaining the matching results, it is necessary to determine whether the matching constitutes a meaningful association based on the preset association determination threshold.

[0190] The association determination threshold is a predefined set of numerical conditions, which usually includes the minimum number of matches threshold and the minimum matching ratio threshold.

[0191] For example, it might be required that at least 10 records be matched, and that the matched values ​​represent more than 60% of the total number of non-null values ​​in the non-primary key columns of the change table. The system compares the count and percentage values ​​in the matching results with the corresponding thresholds.

[0192] The system determines that there is a statistically significant reference relationship between the two columns being compared only when the matching results simultaneously meet or exceed all preset threshold conditions.

[0193] When the above conditions are met, the system determines a candidate association. A candidate association is a structured assertion that records the source table (i.e., the first type of data table), the source column (the non-primary key column being compared), the target table (i.e., the second type of data table), and the target column (the primary key column).

[0194] Iterate through all combinations of non-primary key columns from the first-type data tables and primary key columns from all second-type data tables, repeatedly performing the aforementioned feature subset extraction, intersection operation, and threshold determination process. Collect all candidate relationships determined by each satisfied condition, summarizing them into a candidate relationship set containing multiple such relationships. This set represents the complete discovery result of this round of incremental calculation for all newly added, potential inter-table reference relationships.

[0195] By using set operations and threshold judgment steps, it is possible to efficiently and accurately locate newly generated relationships from changed data without having to perform pairwise comparisons of the entire database.

[0196] In some embodiments, the association determination threshold is optionally calculated dynamically based on the data characteristics of the non-primary key columns and the primary key columns involved in the comparison. The system analyzes the cardinality and value distribution of the non-primary key columns, as well as the cardinality of the target primary key column, and dynamically generates quantity and proportion thresholds suitable for the current column pair matching scenario based on a statistical model. For example, for columns with a large cardinality, the proportion threshold can be appropriately lowered, while for columns with a small cardinality, a higher matching proportion is required. This dynamic threshold mechanism enables association determination to adapt to different data distributions, improving the accuracy and flexibility of relationship discovery.

[0197] In some embodiments, optionally, multiple different subsets of non-primary key and primary key features are assigned to different computing threads or distributed computing nodes, and multiple independent intersection operations are performed simultaneously. After each computing unit returns its matching result, the master node summarizes and determines the threshold. Parallel computing significantly shortens the time required for full feature collision in a large-scale database environment, improving the overall processing throughput.

[0198] In some embodiments, optionally, after determining candidate associations, the system performs a semantic consistency verification. Semantic consistency verification checks the column names, data types, and whether there are known naming patterns associated between the source non-primary key column and the target primary key column in historical metadata. Only when the data matching result meets the threshold and passes the semantic consistency verification will the association be ultimately included in the candidate association set, improving the interpretability of the results.

[0199] In one specific embodiment, the present invention optionally proposes an incremental update mechanism of "state fingerprint driving and skeleton caching":

[0200] 1. Definition and maintenance of state fingerprints:

[0201] Define a "state fingerprint" for each table in the database. The fingerprint is composed of the following lightweight metadata: row count (Row_Count), maximum primary key value (Max_ID), last update time (Update_Time), and column structure hash (Schema_Hash).

[0202] The system maintains a local state file to persistently store the state fingerprints of all tables from the last scan.

[0203] 2. Incremental Change Detection Algorithm:

[0204] Each time a task starts, a lightweight metadata scan is performed first to obtain the real-time status fingerprints of all tables in the current database.

[0205] The real-time fingerprint is compared with the local state file. If any fingerprint indicator of a table changes (such as Max_ID increasing), it is marked as a "Dirty Table"; otherwise, it is marked as a "Clean Table".

[0206] 3. Differentiated scanning strategy:

[0207] For dirty tables: Perform a full / deep sampling scan to read their actual data content and update their detailed data characteristics (such as column value sets).

[0208] For clean tables: Skip data reading and load their "skeleton fingerprint" (containing only key features for association matching, such as the primary key set) directly from the local cache or history, without performing database input / output (I / O) operations.

[0209] 4. Related calculations and state synchronization:

[0210] The correlation is calculated by comparing the newly scanned dirty table data with the loaded clean table skeleton data.

[0211] Once the task is completed, the latest state fingerprint of the dirty table is updated to the local state file, thus completing the closed loop.

[0212] Specifically, skeleton caching is not just about storing data locally, but rather a proactive memory downgrade and on-demand retrieval mechanism.

[0213] The differentiated scanning strategy in this plan specifically includes two dimensions:

[0214] (a) Lightweight probes for execution dimensions (Dry Run / pk_only mode): For tables that the system determines to be clean (unchanged), the system does not perform a full table scan of SELECT *, but only issues query commands for the primary key column (such as SELECT {pk} FROM {tbl} LIMIT 30 in the code), which greatly saves network bandwidth and database I / O.

[0215] (b) Memory Downgrade Self-Release: For dirty tables that have completed a full scan, after extracting data features (Sample DataFrame) and completing the current batch calculation, the system actively performs memory downgrade (corresponding to the code that only retains valid_pks and cleans up other columns, triggering gc.collect()). That is, it releases all sample data of non-primary key columns, retaining only the primary key fingerprint (PK Samples) in memory as the skeleton.

[0216] When incremental updates occur, the association calculation adopts an asymmetric hybrid collision model of full dirty table samples and skeleton net table fingerprints.

[0217] The specific steps are as follows:

[0218] (a) The non-primary key column fingerprint set of the dirty table (such as samples extracted from newly added business data) is intersected with the primary key fingerprint set of the clean table (Global PK Samples) in the global cache in memory.

[0219] (b) If the number of intersection hits (matched_count) is greater than the minimum number of matches and the percentage is greater than the matching threshold (e.g., 0.6), then it is determined that the data added to the dirty table references the data in the clean table that has not been changed.

[0220] (c) State Synchronization and Linked Updates: When a new foreign key outflow is detected in a dirty table and points to an old table (clean table), the system adds the clean table to the list of affected linked updates (affected_tables_map). If only the skeleton of the clean table remains in memory, the system triggers a targeted fetch operation to restore its full sample to reconstruct the complete semantic ontology (RDF / TTL), and finally serializes the merged state fingerprint into the state file (JSON).

[0221] Understandably, for tables that remain unchanged (which usually account for the vast majority), I / O overhead is reduced to zero, with only a small amount of memory consumed to load the skeleton, resulting in an overall processing speed increase of several to tens of times.

[0222] Furthermore, by loading the skeleton fingerprint of the net table, it is ensured that the relationships of "new table referencing old table" or "old table referencing new table" are not missed due to incremental scanning.

[0223] In one specific embodiment, the data lake environment may optionally contain a large number of frequently changing business data tables.

[0224] Experimental environment and data:

[0225] Overall Background: This is a comprehensive data asset governance project for an enterprise, involving heterogeneous systems across 15 business areas, including Enterprise Resource Planning (ERP), Customer Relationship Management (CRM), and Human Resources (HR).

[0226] Data source: Contains 12,852 database tables from 14 business systems.

[0227] Data volume: The total number of data rows exceeds 5 billion, with approximately 50 million new data rows added daily.

[0228] Change frequency: Approximately 5% of the table structure or data undergoes substantial changes daily.

[0229] Detection and experimental parameters:

[0230] Status fingerprint sensitivity test:

[0231] Test scenarios: Simulate four changes: "add a row of data", "modify a row of non-indexed data", "add a column", and "delete a column".

[0232] result:

[0233] New data: If Row_Count or Max_ID changes, the system correctly marks it as a "dirty table".

[0234] Data modification: Update_Time has changed, and the system correctly marks it as a "dirty table".

[0235] Structure change: The Schema_Hash has changed, and the system correctly marks it as a "dirty table".

[0236] Conclusion: State fingerprinting can cover 100% of all changes that affect association calculations.

[0237] Incremental update efficiency comparison:

[0238] Scenario: In an environment containing 12,000 tables, simulate changes to 600 tables (approximately 5%).

[0239] Improvements: 22x faster time efficiency and 93% lower I / O overhead.

[0240] Skeleton cache validity verification:

[0241] Scenario: The primary key of table A (unchanged) is referenced by the foreign key of table B (newly changed).

[0242] Result: Although the system skipped the scan of table A, it still successfully calculated the association between table B and table A by loading the local skeleton fingerprint.

[0243] The recall rate for association discovery remained above 90%, with no relationships lost due to incremental skipping.

[0244] Experiments have shown that the data association increment determination method greatly reduces the consumption of computing resources while ensuring the accuracy of association calculations, making it possible to perform daily or even hourly full-domain association updates for large-scale databases.

[0245] In one specific embodiment, optionally, the schematic diagram of the algorithm corresponding to the data association increment determination method provided by the present invention is as follows: Figure 7 As shown, the specific steps include:

[0246] Fingerprint extraction and alteration detection:

[0247] The process begins with task initiation. Subsequently, the system executes two lightweight operations in parallel to obtain the necessary comparison benchmarks. On one hand, a lightweight metadata scan is performed, with the corresponding method being get_db_table_features. This operation quickly retrieves real-time metadata from all tables in the target database and forms a current state fingerprint. The current state fingerprint includes key features such as the number of rows (Row_Count), the maximum value of the primary key (Max_ID), and the structure hash value (Hash).

[0248] On the other hand, the local state file is loaded, and the corresponding method is load_history_state, which reads the "historical state fingerprint" that was persisted after the last successful execution of the task from the local disk.

[0249] Next, the system enters the fingerprint comparison stage, using the detect_changes method.

[0250] In this step, the system compares the "current state fingerprint" with the "historical state fingerprint" of each data table item by item. Based on the comparison results, the process generates a unique branch judgment: if any inconsistency is found in the fingerprint, it is judged as "fingerprint change"; if the fingerprints are completely consistent, it is judged as "fingerprint consistency".

[0251] Differentiated scanning strategy:

[0252] Based on the branching decision, the system adopts completely different processing strategies for the data table, forming two paths.

[0253] Handling of Dirty Tables: Tables marked with "fingerprint changes" are called "dirty tables".

[0254] The system performs resource-intensive "DB I / O" operations on such tables, specifically "deep sampling scans," executing queries to obtain the latest actual data content. This process outputs the latest sample data, which contains all or representative rows of data from the table, for subsequent in-depth analysis.

[0255] Processing of Clean Tables (green path): Tables marked with "fingerprint consistency" are "clean tables". For these tables, the system skips all database queries and directly performs a "load skeleton cache" operation, such as loading historical data from a local JSON file. This process outputs "historical skeleton data," which contains only primary key column information. This means the system directly "reuses" historical calculation results, avoiding any database I / O overhead.

[0256] Hybrid computing and closed-loop:

[0257] The outputs from both paths in the step, namely the "latest sample data" from the dirty table and the "historical skeleton data" from the net table, are placed together into the "hybrid memory pool".

[0258] Based on this hybrid dataset, the system performs core association calculations and collision detection. It uses asymmetric comparisons of the full sample data from dirty tables with the primary key skeletons of clean tables to discover newly added or changed inter-table relationships. Finally, the system performs an "update state file and persist" operation. This operation writes the latest state fingerprints (especially the new fingerprints of dirty tables) calculated in this scan back to the local state file, completing the loop of this incremental task and providing a new comparison benchmark for the next task.

[0259] In one specific embodiment, optionally, the state fingerprint structure and determination logic include: the generation of the state fingerprint, i.e., the metadata combination (StateFingerprint), is implemented at the code level by the get_db_table_features function, wherein the value of the schema_hash attribute is obtained by performing a hash operation on the combination of the column name (col_name) and data type (type) of the table.

[0260] For example, the corresponding code is: get_db_table_features, MD5(col_name+type), schema_hash.

[0261] Metadata combinations include, but are not limited to:

[0262] +int row_count: Represents a public integer attribute used to record the number of rows in a data table.

[0263] +string max_id: Represents a public string attribute used to record the maximum value of the primary key column of a data table.

[0264] +string update_time: Represents a public string attribute used to record the last update time of the data table.

[0265] +string schema_hash: Represents a public string attribute used to store the hash value of the data table structure definition, which is a unique digest of the table structure.

[0266] Internal decision-making rules include, but are not limited to:

[0267] 1. Determine if the table structure has changed by checking the Schema_Hash: Check if the current and historical schema hash values ​​are different. If they are different, the table structure has been changed.

[0268] 2. Determine if new data has been added by checking Max_ID: Check if the current maximum ID is greater than the historical value. If so, it is determined that new data has been added to the table.

[0269] 3. Determine if data deletion / insertion occurred by checking Row_Count: Check if the number of rows fluctuated (increased or decreased). If so, it indicates that a data deletion or insertion operation occurred in the table.

[0270] The decision is output based on the results of the internal judgment rules to trigger the ActionTrigger:

[0271] Internal action options:

[0272] Clean: continue / load_history: If the result is "clean" or "unchanged", the action to be performed is to continue the subsequent process or directly load historical data / features without deep processing.

[0273] Dirty: to process.append(tbl): If the result is "dirty" or "modified", the action is to add the table (tbl) to a pending list (to_process) for subsequent deep scan and analysis.

[0274] In one specific embodiment, optionally, the logical diagram of the memory accommodating new data and the old skeleton is as follows: Figure 8 As shown, the database server contains two sample tables that need to be processed.

[0275] Table A is marked as "new data," meaning it was determined by the status fingerprint to be a "dirty table" that has been modified during this incremental cycle. Table B is marked as "no change," meaning it was determined to be a "clean table" that has not been modified.

[0276] Local disk: Stores the History JSON file, which is a persistent cache of historical state fingerprints and features.

[0277] Based on the above status determination, the system performs differentiated data acquisition operations:

[0278] For a dirty table (Table A), the system initiates a Heavy I / O (Select) operation, which involves performing a full table scan or deep sampling to transfer the complete "new data" portion of the table to the processing memory. This is a resource-intensive data path.

[0279] For the clean table (Table B), the system performs only a lightweight metadata scan to confirm its status, and then reads its historical characteristics directly from the History JSON cache through a Fast Load operation, consuming almost no database I / O resources. This is a low-resource-consumption data path.

[0280] The processing memory (RAM) shows how the loaded data is organized, divided into two logical areas, corresponding to two types of data:

[0281] Dirty Data Area (Complete Sample): Complete data from Table A is loaded here, forming a Table A DataSample DataFrame. The data frame contains all columns, such as id, name, ref, id..., and is a detailed collection of data used for correlation matching.

[0282] Skeleton cache (primary key only): Historical features from Table B are loaded here to form Table BSkeleton, which contains only the primary key set (PK Set: {101, 102, 103...}) and no other business columns. This is the embodiment of the meaning of "skeleton", that is, only retaining the minimum necessary information to identify itself and for others to reference.

[0283] Once the two types of data are ready in memory, the system performs the core associative computation. The computational model is "asymmetric":

[0284] The calculation is driven by a question: "Does A reference B?", which means probing whether a column in dirty table Table A references the primary key of clean table Table B.

[0285] Candidate references: The skeleton (PK Set) of the net table Table B serves as the target set for references, awaiting matching.

[0286] Reference source data: The complete sample data of dirty table Table A (especially foreign key candidate columns such as ref and id) serves as the source of the reference.

[0287] The calculation process involves performing a quick intersection operation or membership relationship determination on the set of values ​​in columns such as ref and id of Table A and the set of primary keys in the skeleton of Table B, thereby discovering or verifying the association relationship.

[0288] like Figure 5As shown in the illustration, this application embodiment also provides a data association incremental determination device 900, including: a fingerprint acquisition module 902, used to acquire current state fingerprints corresponding to multiple data tables in a target database, wherein the target database is configured with local cache data and local state data; a vector determination module 904, used to determine a state change feature vector corresponding to each data table based on the current state fingerprint and the local state data; a data classification module 906, used to determine at least one first type data table and at least one second type data table from the multiple data tables based on the state change feature vector; and a full scan module 908, used to determine a detailed data feature set based on at least one first type data table. The detailed data feature set includes at least the updated status fingerprint and non-primary key column value features; the cache loading module 910 is used to obtain the historical key data feature set corresponding to each second type of data table from the local cache data, and the historical key data feature set includes at least the primary key column value features and / or the primary key fingerprint generated based on the primary key column value features; the association calculation module 912 is used to determine the candidate association relationship set based on the detailed data feature set and the historical key data feature set; the incremental determination module 914 is used to determine the association incremental result corresponding to the first type of data table based on the candidate association relationship set; and the status synchronization module 916 is used to update the local status data based on the association incremental result.

[0289] like Figure 6 As shown, this application embodiment also provides an electronic device 1000, including a processor 1110, a memory 1109, and a program or instructions stored in the memory 1109 and executable on the processor 1110. When the program or instructions are executed by the processor 1110, they implement the various processes of the above-described data association increment determination method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0290] Optionally, the processor 1110 is used to obtain the status fingerprints of multiple data tables in the target database, the data tables including local cached data, and the target database has corresponding local status data set;

[0291] Optionally, the processor 1110 is also configured to determine the state change feature vector corresponding to each data table based on the state fingerprint and local state data.

[0292] Optionally, the processor 1110 is also configured to determine at least one first type data table and at least one second type data table from multiple data tables based on the state change feature vector;

[0293] Optionally, the processor 1110 is further configured to determine a detailed data feature set based on at least one first type data table, the detailed data feature set including at least one updated state fingerprint;

[0294] Optionally, the processor 1110 is also configured to obtain a set of historical key data features corresponding to each second type of data table from local cache data, the set of historical key data features including at least one primary key fingerprint;

[0295] Optionally, the processor 1110 is further configured to determine a candidate association set based on at least one updated state fingerprint in a detailed data feature set and at least one primary key fingerprint in a historical key data feature set;

[0296] Optionally, the processor 1110 is further configured to determine the incremental association result corresponding to the first type of data table based on the candidate association set;

[0297] Optionally, the processor 1110 is also used to update local state data based on the associated incremental results.

[0298] The memory 1109 can be used to store software programs and various data. The memory 1109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1109 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0299] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described data association increment determination method embodiments and achieve the same technical effects. To avoid repetition, these will not be described again here. Furthermore, the readable storage medium improves the data storage capacity and data processing speed corresponding to the data association increment determination method in this application.

[0300] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. A computer-readable storage medium can be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing, but is not limited thereto. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital universal disk (DVD), memory cards, floppy disks, encoding mechanical devices (e.g., punched cards or grooves with raised structures for recording instructions), and any suitable combination of the foregoing. The computer-readable storage medium used herein should not be construed as the transmission of signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media, or electrical signals transmitted through wires.

[0301] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0302] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described data association increment determination method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here. In addition, the chip improves the data processing speed corresponding to the data association increment determination method in this application.

[0303] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0304] In this invention, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance; the term "multiple" refers to two or more unless otherwise explicitly defined. The terms "install," "connect," "link," and "fix" should be interpreted broadly. For example, "connect" can be a fixed connection, a detachable connection, or an integral connection; "link" can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0305] In the description of this invention, it should be understood that the terms "upper," "lower," "left," "right," "front," "rear," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or unit referred to must have a specific orientation or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0306] In the description of this specification, the terms "one embodiment," "some embodiments," "specific embodiment," etc., refer to specific features, structures, materials, or characteristics described in connection with an embodiment or example that are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0307] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for determining data association increments, characterized in that, include: Obtain the current state fingerprints corresponding to multiple data tables in the target database, wherein the target database is configured with local cache data and local state data. Determine the state change feature vector corresponding to each of the data tables based on the current state fingerprint and the local state data; Based on the state change feature vector, at least one first type data table and at least one second type data table are determined from the plurality of data tables; A detailed data feature set is determined based on at least one of the first type of data tables, the detailed data feature set including at least the updated state fingerprint and non-primary key column value features; Obtain a set of historical key data features corresponding to each of the second type of data tables from the local cache data. The set of historical key data features includes at least primary key column value features and / or primary key fingerprints generated based on the primary key column value features. A candidate association set is determined based on the detailed data feature set and the historical key data feature set; The incremental association results corresponding to the first type of data table are determined based on the candidate association set; The local state data is updated based on the correlation increment results.

2. The data association increment determination method according to claim 1, characterized in that, The step of determining the state change feature vector corresponding to each data table based on the current state fingerprint and the local state data includes: Obtain the historical state fingerprint corresponding to each data table from the local state data; For each data table, multiple feature values ​​in the current state fingerprint are compared with multiple historical feature values ​​in the historical state fingerprint to determine the first comparison result; Based on the first comparison result, determine the state change feature vector used to characterize whether the data table has been changed and the dimension of the change; The current state fingerprint contains multiple feature values, including row count feature value, primary key maximum value feature value, last update time feature value, and table structure hash feature value.

3. The data association increment determination method according to claim 2, characterized in that, The step of determining at least one first-type data table and at least one second-type data table from multiple data tables based on the state change feature vector includes: Each state change feature vector is compared with a preset change determination threshold to determine a second comparison result; Based on the second comparison result, the data table whose change amount corresponding to the feature value exceeds the corresponding change judgment threshold is determined as the first type of data table; Data tables whose changes corresponding to the feature values ​​do not exceed the corresponding change determination threshold are identified as second type data tables.

4. The data association increment determination method according to claim 3, characterized in that, The step of determining a detailed set of data features based on at least one of the first type of data tables includes: Perform a full table scan or deep sampling scan on the first type of data table to obtain full data or sampled data, and extract data features containing non-primary key column value features from the full data or sampled data to form the detailed data feature set.

5. The data association increment determination method according to claim 3, characterized in that, The step of obtaining the set of historical key data features corresponding to each of the second type of data tables from the local cache data includes: For the second type of data table, perform a lightweight query or read directly from the cache to obtain the primary key column value characteristics, forming the historical key data feature set.

6. The method for determining data association increments according to any one of claims 1 to 5, characterized in that, The step of determining the candidate association set based on the detailed data feature set and the historical key data feature set includes: Based on the detailed data feature set, at least one subset of non-primary key column value features is determined; Determine at least one subset of primary key column value features based on the aforementioned set of historical key data features; Perform a matching analysis between the non-primary key column value feature subset and the primary key column value feature subset to determine the matching result; At least one candidate association relationship is determined based on whether the matching result meets a preset association determination threshold; The candidate association set is determined based on multiple candidate associations.

7. A data association increment determination device, characterized in that, include: The fingerprint acquisition module is used to acquire the current state fingerprints corresponding to multiple data tables in the target database, wherein the target database is configured with local cache data and local state data. The vector determination module is used to determine the state change feature vector corresponding to each of the data tables based on the current state fingerprint and the local state data. The data classification module is used to determine at least one first-type data table and at least one second-type data table from multiple data tables based on the state change feature vector; A full scan module is used to determine a detailed data feature set based on at least one of the first type of data tables, the detailed data feature set including at least the updated state fingerprint and non-primary key column value features; A cache loading module is used to obtain a set of historical key data features corresponding to each of the second type of data tables from the local cache data. The set of historical key data features includes at least primary key column value features and / or primary key fingerprints generated based on the primary key column value features. The association calculation module is used to determine a candidate association set based on the detailed data feature set and the historical key data feature set. The incremental determination module is used to determine the incremental association result corresponding to the first type of data table based on the candidate association set; The status synchronization module is used to update the local status data based on the associated incremental results.

8. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the data association increment determination method as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the data association increment determination method as described in any one of claims 1 to 6.

10. A chip, characterized in that, The chip includes a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the data association incremental determination method as described in any one of claims 1 to 6.

Images