Table-level data classification and grading method and system based on big data

Abstract

The invention discloses a table-level data classification and grading method and system based on big data, and relates to the technical field of table-level data classification and grading. According to the scheme, second-level sensitive table grading response is achieved through streaming incremental calculation and dynamic semantic coupling; a finite-state machine and a Hash fingerprint technology accurately capture table structure and content mutation, and traditional batch processing delay is avoided; the multi-modal fusion model synchronously analyzes topological features and semantic vectors, and the field-level sensitivity misjudgment problem is solved; the timeliness weight mechanism guarantees real-time evaluation of a dynamic association table relationship, and is particularly suitable for a high-frequency change scene in a big promotion period; and the asynchronous architecture design isolates the calculation load, so that the processing priority of the core price list is effectively guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of table-level data classification and grading, and in particular to a table-level data classification and grading method and system based on big data. Background Art

[0002] In e-commerce promotion scenarios, price strategy tables are highly dynamic and sensitive. These tables carry core business information such as flash sale prices and purchase discount rules. Their field values ​​may be updated dozens of times per second and are strongly correlated with inventory and user behavior tables. Existing mainstream solutions use dynamic metadata graph technology to track table structure changes and classify them, introducing graph neural networks (GNNs) to capture cross-table associations. Traditional solutions suffer from real-time gaps, lack of semantic understanding, and failure of dynamic associations during peak promotion periods.

[0003] Although existing dynamic graph solutions can perceive changes in table structure, they have difficulty coping with sudden changes at the field content level. For example, the moment the "discount rate" field in the pricing strategy table switches from an estimated value to a real-time value, data sensitivity increases dramatically. However, traditional feature extraction relies on table structure metadata and cannot capture the semantic transition of field values. At the same time, the batch-based GNN model produces minute-level delays when calculating large-scale table relationships, resulting in the inability to promptly include newly added related tables in the grading evaluation. More importantly, the existing solution lacks fine-grained semantic understanding capabilities, treating common fields such as user_rating the same as the sensitive field payment_token, resulting in overly coarse grading granularity.

[0004] To this end, some solutions adopt a hybrid computing architecture that combines streaming computing engines (such as Flink) with lightweight GNN models to shorten response delays through incremental feature updates; some solutions introduce field name semantic analysis modules and use pre-trained word vectors to distinguish sensitive fields. However, the problem of detecting sudden changes in numerical distribution has not yet been solved and still needs to rely on manual rules. Summary of the Invention

[0005] In view of the above existing problems, the present invention is proposed.

[0006] The present invention provides a table-level data classification and grading method and system based on big data to solve the three major bottlenecks faced by existing dynamic graph solutions during big promotions: insufficient real-time performance, lack of semantic understanding, and failure of dynamic association, and difficulty in coping with instantaneous sensitivity changes at the field level.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In a first aspect, an embodiment of the present invention provides a table-level data classification and grading method based on big data, which includes:

[0009] Step S1: collecting the log change stream of the target database in real time based on the stream computing engine, wherein the log change stream includes data definition language operation records and data manipulation language operation records;

[0010] Step S2, performing real-time analysis on the log change stream by an incremental computing unit to extract table structure change features and field content distribution features;

[0011] Step S3: calling a semantic analysis module to perform dynamic semantic enhancement on the field content distribution characteristics to generate a field sensitivity vector;

[0012] Step S4: construct a dynamic metadata graph, which includes table nodes, field nodes and their associated relationship weights;

[0013] Step S5: Using a multimodal fusion model, the table structure change features, field sensitivity vectors, and dynamic metadata maps are input to generate table classification and grading results.

[0014] As a preferred solution of the table-level data classification and grading method based on big data described in the present invention, the incremental calculation unit includes:

[0015] The schema change detection subunit identifies the DDL operation type through a finite state machine;

[0016] The content feature extraction subunit uses hash fingerprint comparison technology to detect changes in field value distribution. When the hash fingerprint difference of field values ​​in N consecutive time windows exceeds a threshold, it is marked as a distribution shift event.

[0017] The operations performed by the incremental calculation unit include:

[0018] When a data definition language operation is detected, the local subgraph of the dynamic metadata graph is updated;

[0019] For data manipulation language operations, a sliding time window is used to calculate the distribution deviation of field values;

[0020] The association relationship weight is dynamically adjusted according to a preset weight decay function, where the decay function is negatively correlated with the time interval.

[0021] As a preferred solution of the table-level data classification and grading method based on big data described in the present invention, in step S2, during the process of identifying the DDL operation type through a finite state machine, a finite state machine is constructed:

[0022] M=(Q,Σ,δ,q0,F),

[0023] Where Q represents the state set, Σ represents the input symbol set, δ represents the transition function, which is defined as δ:Q×Σ→Q, q0 represents the initial state, and F represents the set of accepted states, corresponding to different DDL types;

[0024] When the log words are input into M one by one, if the terminal state q end ∈F, then based on q end To classify:

[0025]

[0026] Among them, q end Indicates the terminal state of the machine after the log word is input, q cr ,q al and q dr Indicates the acceptance status of the corresponding DDL operation. Each acceptance status corresponds to a unique DDL type.

[0027] In step S2, hash fingerprint comparison technology is used to detect changes in field value distribution. In each time window, a multi-path fingerprint vector is constructed for the field value set:

[0028]

[0029] Where i represents the time window index, K represents the number of hash functions, and f i,k represents the k-th hash fingerprint value, D i Represents the field value set in window i, v represents a field value in the set, hash k represents the kth hash function, B represents the modulus, which is used to compress the fingerprint range;

[0030] The L1 distance between adjacent fingerprint vectors is defined as:

[0031] d i =||f i -f i-1 ||1,

[0032] Among them, d i represents the distribution difference between window i and the previous window, ‖·‖1 represents the L1 norm of the vector;

[0033] If d is satisfied in the most recent N consecutive windows j >θ,j=i-N+1,…,i, then the distribution shift event is marked, where N is the number of consecutive windows, θ is the difference threshold, and j is the traversal index;

[0034] In step S2, the method of using a sliding time window to calculate the distribution deviation of the field value is as follows:

[0035] Introduce a sliding window of length L and calculate the average deviation:

[0036]

[0037] Among them, t represents the current window index, L represents the sliding window length, k represents the offset of the offset difference within the window, and d t-k represents the distribution difference of the tkth window;

[0038] When S t When >τ, it indicates that the overall field distribution has shifted significantly in the recent period, where τ represents the average deviation threshold.

[0039] As a preferred solution of the table-level data classification and grading method based on big data described in the present invention, the operation of the semantic analysis module includes:

[0040] Load the pre-trained domain-sensitive vocabulary and perform vectorized encoding on the field names;

[0041] Combine the numerical distribution characteristics of field values ​​with the field name vector to generate a composite semantic vector;

[0042] When a sudden change in field value distribution is detected, the sensitivity weight of the corresponding field is automatically increased;

[0043] The composite semantic vector generation includes:

[0044] Calculate the kurtosis and skewness coefficients within the sliding window for numeric fields and generate statistical feature vectors;

[0045] After extracting the root word of the field name, match it with the domain-sensitive vocabulary to generate a word weight vector;

[0046] Perform Hadamard product operation on the statistical feature vector and the word weight vector to output a composite semantic vector.

[0047] As a preferred solution of the table-level data classification and grading method based on big data described in the present invention, in the process of generating the composite semantic vector in step S3, after preprocessing the field name, its root set R = {r1, r2, ..., r m}, and let the domain sensitive vocabulary

[0048] For each component k∈{κ,γ} of the statistical eigenvector, define the weight:

[0049] like but otherwise

[0050] w=(β κ ,β γ ), where m represents the total number of word roots after field name segmentation, rj represents the jth root, n represents the total number of entries in the sensitive vocabulary, l i represents the root of the i-th sensitive word, π(r j ) represents the mapping function, which will match the root r j Locate the corresponding sensitive vocabulary entry index, Indicates the πth (r j ) root word corresponding to the sensitivity weight of component k, represents the default basic weight of component k when there is no match, κ represents the component corresponding to the kurtosis coefficient, γ represents the component corresponding to the skewness coefficient, and w represents the final word weight vector;

[0051] Then the calculated statistical eigenvector t=(t κ ,t γ ) is multiplied element-by-element with the word weight vector w to obtain a composite semantic vector:

[0052] h=t⊙w=(t κ β κ ,t γ β γ ),

[0053] Among them, t represents the statistical characteristic vector, including the kurtosis coefficient t κ and skewness coefficient t γ , w represents the word weight vector, including the corresponding component weight β κ and β γ , ⊙ represents the Hadamard product operation, and h represents the output composite semantic vector.

[0054] As a preferred solution of the table-level data classification and grading method based on big data described in the present invention, the multimodal fusion model includes:

[0055] The structural feature extraction branch captures the topological features of table nodes based on the graph attention network;

[0056] The semantic feature extraction branch captures field-level temporal semantics based on a bidirectional gated recurrent unit;

[0057] The cross-modal interaction layer fuses the topological features and temporal semantics through an adaptive weight distribution mechanism;

[0058] The adaptive weight allocation mechanism includes:

[0059] Apply a spatial attention weight matrix to the topological features;

[0060] Applying temporal attention weight matrix to temporal semantics;

[0061] Dynamically fuse two types of weight matrices through learnable parameters.

[0062] As a preferred solution of the table-level data classification and grading method based on big data described in the present invention, the method adopts an asynchronous processing architecture:

[0063] Set up a high-frequency feature processing channel and use memory-resident computing for a preset set of sensitive fields;

[0064] Set up a batch feature processing channel and use offline batch processing for non-sensitive fields;

[0065] A lock-free ring buffer is deployed in the high-frequency feature processing channel to temporarily store concurrent feature update requests.

[0066] In a second aspect, the present invention provides a table-level data classification and grading system based on big data, comprising:

[0067] Streaming data collection module, used to capture database log change events in real time;

[0068] A dynamic graph calculation module, connected to the streaming data acquisition module, is used to maintain a table relationship graph with time-sensitive weights;

[0069] Semantic enhancement module with built-in domain-sensitive vocabulary and distribution shift detection unit;

[0070] Asynchronous processing engine, including independent high-frequency processing unit and batch processing unit;

[0071] Hierarchical decision-making module, integrating multimodal neural network and rule reasoner.

[0072] As a preferred solution of the table-level data classification and grading system based on big data described in the present invention, the dynamic graph calculation module includes:

[0073] a subgraph difference calculation unit, generating a local graph update instruction in response to a data definition language operation;

[0074] The time-sensitive weight regulator dynamically decays the edge weight according to the latest access time of the association table;

[0075] The implicit relationship reasoning unit infers potential table associations based on three-degree association path analysis.

[0076] As a preferred solution of the table-level data classification and grading system based on big data described in the present invention, in which: in the dynamic graph calculation module, the associated edge e between any table nodes i and j is ij , its weight decays over time through an exponential decay function, the formula is:

[0077] Among them, w ij (t) represents the current time t below e ijThe weight of t0 represents the reference time of the last calculation or update, w ij (t0) represents the stored weight value at time t0, λ represents the time attenuation coefficient, which is used to control the attenuation speed, Indicates the timestamp when tables i and j were last accessed or updated together. That is, Δt ij , represents the time interval since the edge was last visited;

[0078] Each time a table is accessed or metadata is updated, the current time t is read first and w is recalculated. ij (t), then write t into And with this w ij (t) is the new benchmark weight w ij (t0).

[0079] The beneficial effects of the present invention are as follows: this solution achieves a hierarchical response to sensitive tables within seconds through the coupling of streaming incremental computing and dynamic semantics; the finite state machine and hash fingerprint technology accurately capture table structure and content mutations, avoiding traditional batch processing delays; the multimodal fusion model synchronously analyzes topological features and semantic vectors to solve the problem of field-level sensitivity misjudgment; the timeliness weight mechanism ensures real-time evaluation of dynamic association table relationships, which is particularly suitable for high-frequency change scenarios during major promotions; the asynchronous architecture design isolates computing loads and effectively guarantees the processing priority of core price tables. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0081] Figure 1 This is a flow chart of the table-level data classification and grading method based on big data in Example 1.

[0082] Figure 2 This is a schematic diagram of the framework of the table-level data classification and grading system based on big data in Example 1. DETAILED DESCRIPTION

[0083] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0084] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0085] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0086] Example 1, with reference to Figure 1 and Figure 2 This embodiment provides a table-level data classification and grading method based on big data, including the following steps:

[0087] Step S1: collecting the log change stream of the target database in real time based on the stream computing engine. The log change stream includes data definition language operation records and data manipulation language operation records.

[0088] Step S2: parse the log change stream in real time using an incremental computing unit to extract table structure change features and field content distribution features;

[0089] The incremental calculation unit includes:

[0090] The schema change detection subunit uses a finite state machine to identify DDL operation types (including CREATE / ALTER / DROP);

[0091] The content feature extraction subunit uses hash fingerprint comparison technology to detect changes in field value distribution. When the hash fingerprint difference of field values ​​in N consecutive time windows exceeds a threshold, it is marked as a distribution shift event.

[0092] The operations performed by the incremental calculation unit include:

[0093] When a data definition language operation is detected, the local subgraph of the dynamic metadata graph is updated;

[0094] For data manipulation language operations, a sliding time window is used to calculate the distribution deviation of field values;

[0095] Dynamically adjust the association weight according to the preset weight decay function, and the decay function is negatively correlated with the time interval;

[0096] In step S2, during the process of identifying the DDL operation type through the finite state machine, a finite state machine is constructed:

[0097] M=(Q,Σ,δ,q0,F),

[0098] Where Q represents the state set, Σ represents the input symbol set, δ represents the transition function, which is defined as δ:Q×Σ→Q, q0 represents the initial state, and F represents the set of accepted states, corresponding to different DDL types;

[0099] When the log words are input into M one by one, if the terminal state q end ∈F, then based on q end To classify:

[0100]

[0101] Among them, q end Indicates the terminal state of the machine after the log word is input, q cr ,q al and q dr Indicates the acceptance status of the corresponding DDL operation. Each acceptance status corresponds to a unique DDL type.

[0102] This design leverages the memoryless migration characteristics of finite state automata to achieve constant-time classification of various DDL operations in SQL logs. The strict definition of the state set Q and the symbol set Σ ensures that different keywords do not cause ambiguity. Each element of the accepted state set F corresponds to an operation type, which can directly drive subsequent metadata updates. Since only the current state needs to be tracked, there is no need to save historical logs. The overall space complexity is O(|Q| + |Σ|), which can handle high-concurrency log parsing tasks and support the subsequent expansion of more DDL types.

[0103] In step S2, hash fingerprint comparison technology is used to detect changes in field value distribution. In each time window, a multi-path fingerprint vector is constructed for the field value set:

[0104]

[0105] Where i represents the time window index, K represents the number of hash functions, and f i,k represents the k-th hash fingerprint value, D i Represents the field value set in window i, v represents a field value in the set, hash k represents the kth hash function, B represents the modulus, which is used to compress the fingerprint range;

[0106] The L1 distance between adjacent fingerprint vectors is defined as:

[0107] d i =||f i -f i-1 ||1,

[0108] Among them, d irepresents the distribution difference between window i and the previous window, and ||·||1 represents the L1 norm of the vector;

[0109] If d is satisfied in the most recent N consecutive windows j >θ,j=i-N+1,…,i, then the distribution shift event is marked, where N is the number of consecutive windows, θ is the difference threshold, and j is the traversal index;

[0110] Multi-way hash fingerprint can approximate high-dimensional distribution in constant space, control memory overhead through modular operation B, L1 norm d i Sensitive to subtle distribution fluctuations, combined with continuous N-window detection, it can eliminate occasional jitter and trigger alerts only for sustained changes. This method does not require storing the full frequency distribution, only maintaining the two fingerprint vectors and the difference sequence. The computational complexity is linearly related to K. With the adjustability of the threshold θ and the window length N, it can flexibly adapt to the distribution stability requirements of different businesses.

[0111] In step S2, the method of using a sliding time window to calculate the distribution deviation of the field value is as follows:

[0112] Introduce a sliding window of length L and calculate the average deviation:

[0113]

[0114] Among them, t represents the current window index, L represents the sliding window length, k represents the offset of the offset difference within the window, and d t-k represents the distribution difference of the tkth window;

[0115] When S t When >τ, it indicates that the overall shift of the recent field distribution is significant, where τ represents the average shift threshold;

[0116] Sliding average S t It smoothes single-point difference fluctuations and provides a more robust offset measurement. The window length L determines the response speed to historical differences, taking into account both sensitivity to mutations and noise suppression. By tuning the threshold τ, it can distinguish between normal updates and abnormal offsets. This method only requires accumulating and subtracting boundary differences, has an O(1) update overhead, and is suitable for millisecond-level streaming computing scenarios.

[0117] Step S3: calling the semantic analysis module to perform dynamic semantic enhancement on the field content distribution characteristics to generate a field sensitivity vector;

[0118] The operations of the semantic analysis module include:

[0119] Load the pre-trained domain-sensitive vocabulary and perform vectorized encoding on the field names;

[0120] Combine the numerical distribution characteristics of field values ​​with the field name vector to generate a composite semantic vector;

[0121] When a sudden change in field value distribution is detected, the sensitivity weight of the corresponding field is automatically increased;

[0122] Composite semantic vector generation includes:

[0123] Calculate the kurtosis and skewness coefficients within the sliding window for numeric fields and generate statistical feature vectors;

[0124] After extracting the root word of the field name, match it with the domain-sensitive vocabulary to generate a word weight vector;

[0125] Perform Hadamard product operation on the statistical feature vector and the word weight vector to output a composite semantic vector;

[0126] In the process of generating the composite semantic vector in step S3, after preprocessing the field name, its root set R = {r1, r2, ..., r m}, and let the domain sensitive vocabulary

[0127] For each component k∈{κ,γ} of the statistical eigenvector, define the weight:

[0128] like but otherwise

[0129] w=(β κ ,β γ ), where m represents the total number of word roots after field name segmentation, r j represents the jth root, n represents the total number of entries in the sensitive vocabulary, l i represents the root of the i-th sensitive word, π(r j ) represents the mapping function, which will match the root r j Locate the corresponding sensitive vocabulary entry index, Indicates the πth (r j ) root word corresponding to the sensitivity weight of component k, represents the default basic weight of component k when there is no match, κ represents the component corresponding to the kurtosis coefficient, γ represents the component corresponding to the skewness coefficient, and w represents the final word weight vector;

[0130] Specifically, by strictly matching the field noun root with the pre-trained sensitive word library, only the word roots that are deeply associated with the field can affect the sensitivity weight. The maximum pooling strategy max can quickly select the most significant sensitivity among multiple matching items, and set the default weight at the same time. To ensure basic semantic contribution even when there is no match, the parameter mapping function π guarantees the unique correspondence between the root word and the sensitive word library entry. The time complexity of this step is O(m), which is suitable for streaming scenarios and easy to expand: when adding new sensitive words, there is no need to modify the process, only to add That's it;

[0131] Then the calculated statistical eigenvector t=(t κ ,t γ ) is multiplied element-by-element with the word weight vector w to obtain a composite semantic vector:

[0132] h=t⊙w=(t κ β κ ,t γ β γ ),

[0133] Among them, t represents the statistical characteristic vector, including the kurtosis coefficient t κ and skewness coefficient t γ , w represents the word weight vector, including the corresponding component weight β κ and β γ , ⊙ represents the Hadamard product operation, h represents the output composite semantic vector;

[0134] Specifically, through the Hadamard product operation, each component in the statistical feature vector is amplified or suppressed by the corresponding semantic weight, achieving dynamic coupling of changes in field value distribution and domain sensitivity. This operation only requires a constant time O(d) (where d = 2), with extremely low overhead and suitable for online computing scenarios. The composite vector h simultaneously encodes the distribution characteristics of the field within the sliding window and the domain association of the field name, helping downstream multimodal models capture cross-dimensional semantic and statistical interactions, improving the accuracy and reliability of classification and grading.

[0135] Step S4: construct a dynamic metadata graph, which includes table nodes, field nodes and their associated relationship weights;

[0136] Step S5: Using a multimodal fusion model to input table structure change features, field sensitivity vectors, and dynamic metadata maps to generate table classification and grading results;

[0137] Multimodal fusion models include:

[0138] The structural feature extraction branch captures the topological features of table nodes based on the graph attention network;

[0139] The semantic feature extraction branch captures field-level temporal semantics based on a bidirectional gated recurrent unit;

[0140] The cross-modal interaction layer integrates topological features and temporal semantics through an adaptive weight distribution mechanism;

[0141] The adaptive weight distribution mechanism includes:

[0142] Apply a spatial attention weight matrix to the topological features;

[0143] Applying temporal attention weight matrix to temporal semantics;

[0144] Dynamically fuse two types of weight matrices through learnable parameters;

[0145] This method uses an asynchronous processing architecture:

[0146] Set up a high-frequency feature processing channel and use memory-resident computing for a preset set of sensitive fields;

[0147] Set up a batch feature processing channel and use offline batch processing for non-sensitive fields;

[0148] A lock-free ring buffer is deployed in the high-frequency feature processing channel to temporarily store concurrent feature update requests.

[0149] This embodiment also provides a table-level data classification and grading system based on big data, including:

[0150] Streaming data collection module, used to capture database log change events in real time;

[0151] The dynamic graph calculation module is connected to the streaming data acquisition module to maintain the table relationship graph with time-sensitive weights;

[0152] The dynamic graph calculation module includes:

[0153] a subgraph difference calculation unit, generating a local graph update instruction in response to a data definition language operation;

[0154] The time-sensitive weight regulator dynamically decays the edge weight according to the latest access time of the association table;

[0155] Implicit relationship reasoning unit, which infers potential table associations based on three-dimensional association path analysis;

[0156] In the dynamic graph calculation module, for any table node i and j, the associated edge e ij , its weight decays over time through an exponential decay function, the formula is:

[0157] Among them, w ij (t) represents the current time t below e ij The weight of t0 represents the reference time of the last calculation or update, w ij (t0) represents the stored weight value at time t0, λ represents the time attenuation coefficient, which is used to control the attenuation speed, Indicates the timestamp when tables i and j were last accessed or updated together. That is, Δt ij , represents the time interval since the edge was last visited;

[0158] Each time a table is accessed or metadata is updated, the current time t is read first and w is recalculated. ij (t), then write t into And with this w ij (t) is the new benchmark weight w ij (t0);

[0159] Specifically, the exponential decay function can be used to naturally simulate the decay process of edge weights over time, which is more in line with the intuition that the longer the edge has not been accessed, the weaker the association. The coefficient λ can be adjusted according to business needs: the larger the λ, the faster the weight decay, which is suitable for scenarios where hotspot tables are frequently updated. The smaller the λ, the longer the memory of historical accesses is retained, which is suitable for stable association analysis. The last access timestamp is used. Perform differential calculations to avoid traversing all edges. Only the local subgraph being accessed is processed each time to ensure computational efficiency in a streaming environment.

[0160] Semantic enhancement module with built-in domain-sensitive vocabulary and distribution shift detection unit;

[0161] Asynchronous processing engine, including independent high-frequency processing unit and batch processing unit;

[0162] Hierarchical decision-making module, integrating multimodal neural network and rule reasoner.

[0163] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A table-level data classification and grading method based on big data, characterized in that: include, Step S1: collecting the log change stream of the target database in real time based on the stream computing engine, wherein the log change stream includes data definition language operation records and data manipulation language operation records; Step S2, performing real-time analysis on the log change stream by an incremental computing unit to extract table structure change features and field content distribution features; Step S3: calling a semantic analysis module to perform dynamic semantic enhancement on the field content distribution characteristics to generate a field sensitivity vector; Step S4: construct a dynamic metadata graph, which includes table nodes, field nodes and their associated relationship weights; Step S5: Using a multimodal fusion model, the table structure change features, field sensitivity vectors, and dynamic metadata maps are input to generate table classification and grading results.

2. A table-level data classification and grading method based on big data according to claim 1, characterized in that: The incremental calculation unit comprises: The schema change detection subunit identifies the DDL operation type through a finite state machine; The content feature extraction subunit uses hash fingerprint comparison technology to detect changes in field value distribution. When the hash fingerprint difference of field values ​​in N consecutive time windows exceeds a threshold, it is marked as a distribution shift event. The operations performed by the incremental calculation unit include: When a data definition language operation is detected, the local subgraph of the dynamic metadata graph is updated; For data manipulation language operations, a sliding time window is used to calculate the distribution deviation of field values; The association relationship weight is dynamically adjusted according to a preset weight decay function, where the decay function is negatively correlated with the time interval.

3. A table-level data classification and grading method based on big data according to claim 2, characterized in that: In step S2, during the process of identifying the DDL operation type through the finite state machine, a finite state machine is constructed: M=(Q,Σ,δ,q0,F), Where Q represents the state set, Σ represents the input symbol set, δ represents the transition function, which is defined as δ:Q×Σ→Q, q0 represents the initial state, and F represents the set of accepted states, corresponding to different DDL types; When the log words are input into M one by one, if the terminal state q end ∈F, then based on q end To classify: Among them, q end Indicates the terminal state of the machine after the log word is input, q cr ,q al and q dr Indicates the acceptance status of the corresponding DDL operation. Each acceptance status corresponds to a unique DDL type. In step S2, hash fingerprint comparison technology is used to detect changes in field value distribution. In each time window, a multi-path fingerprint vector is constructed for the field value set: f i =(f i,1 ,f i,2 ,…,f i,K ), Where i represents the time window index, K represents the number of hash functions, and f i,k represents the k-th hash fingerprint value, D i Represents the field value set in window i, v represents a field value in the set, hash k represents the kth hash function, B represents the modulus, which is used to compress the fingerprint range; The L1 distance between adjacent fingerprint vectors is defined as: d i =||f i -f i-1 ||1, Among them, d i represents the distribution difference between window i and the previous window, and ||·||1 represents the L1 norm of the vector; If d is satisfied in the most recent N consecutive windows j >θ,j=i-N+1,…,i, then the distribution shift event is marked, where N is the number of consecutive windows, θ is the difference threshold, and j is the traversal index; In step S2, the method of using a sliding time window to calculate the distribution deviation of the field value is as follows: Introduce a sliding window of length L and calculate the average deviation: Among them, t represents the current window index, L represents the sliding window length, k represents the offset of the offset difference within the window, and d t-k represents the distribution difference of the tkth window; When S t When >τ, it indicates that the overall field distribution has shifted significantly in the recent period, where τ represents the average deviation threshold.

4. A table-level data classification and grading method based on big data according to claim 1, characterized in that: The operations of the semantic analysis module include: Load the pre-trained domain-sensitive vocabulary and perform vectorized encoding on the field names; Combine the numerical distribution characteristics of field values ​​with the field name vector to generate a composite semantic vector; When a sudden change in field value distribution is detected, the sensitivity weight of the corresponding field is automatically increased; The composite semantic vector generation includes: Calculate the kurtosis and skewness coefficients within the sliding window for numeric fields and generate statistical feature vectors; After extracting the root word of the field name, match it with the domain-sensitive vocabulary to generate a word weight vector; Perform Hadamard product operation on the statistical feature vector and the word weight vector to output a composite semantic vector.

5. A table-level data classification and grading method based on big data according to claim 4, characterized in that: In the process of generating the composite semantic vector in step S3, after preprocessing the field name, its root set R = {r1, r2, ..., r m }, and let the domain sensitive vocabulary For each component k∈{κ,γ} of the statistical eigenvector, define the weight: like but otherwise w=(β κ ,β γ ), where m represents the total number of word roots after field name segmentation, r j represents the jth root, n represents the total number of entries in the sensitive vocabulary, l i represents the root of the i-th sensitive word, π(r j ) represents the mapping function, which will match the root r j Locate the corresponding sensitive vocabulary entry index, Indicates the πth (r j ) root word corresponding to the sensitivity weight of component k, represents the default basic weight of component k when there is no match, κ represents the component corresponding to the kurtosis coefficient, γ represents the component corresponding to the skewness coefficient, and w represents the final word weight vector; Then the calculated statistical eigenvector t=(t κ ,t γ ) is multiplied element-by-element with the word weight vector w to obtain a composite semantic vector: h=t⊙w=(t κ ·b κ ,t γ ·b γ ), Among them, t represents the statistical characteristic vector, including the kurtosis coefficient t κ and skewness coefficient t γ , w represents the word weight vector, including the corresponding component weight β κ and β γ , ⊙ represents the Hadamard product operation, and h represents the output composite semantic vector.

6. A table-level data classification and grading method based on big data according to claim 1, characterized in that: The multimodal fusion model includes: The structural feature extraction branch captures the topological features of table nodes based on the graph attention network; The semantic feature extraction branch captures field-level temporal semantics based on a bidirectional gated recurrent unit; The cross-modal interaction layer fuses the topological features and temporal semantics through an adaptive weight distribution mechanism; The adaptive weight allocation mechanism includes: Apply a spatial attention weight matrix to the topological features; Applying temporal attention weight matrix to temporal semantics; Dynamically fuse two types of weight matrices through learnable parameters.

7. A table-level data classification and grading method based on big data according to claim 6, characterized in that: This method uses an asynchronous processing architecture: Set up a high-frequency feature processing channel and use memory-resident computing for a preset set of sensitive fields; Set up a batch feature processing channel and use offline batch processing for non-sensitive fields; A lock-free ring buffer is deployed in the high-frequency feature processing channel to temporarily store concurrent feature update requests.

8. A table-level data classification and grading system based on big data, based on a table-level data classification and grading method based on big data according to any one of claims 1 to 7, characterized in that: include: Streaming data collection module, used to capture database log change events in real time; A dynamic graph calculation module, connected to the streaming data acquisition module, is used to maintain a table relationship graph with time-sensitive weights; Semantic enhancement module with built-in domain-sensitive vocabulary and distribution shift detection unit; Asynchronous processing engine, including independent high-frequency processing unit and batch processing unit; Hierarchical decision-making module, integrating multimodal neural network and rule reasoner.

9. A table-level data classification and grading system based on big data according to claim 8, characterized in that: The dynamic graph calculation module includes: a subgraph difference calculation unit, generating a local graph update instruction in response to a data definition language operation; The time-sensitive weight regulator dynamically decays the edge weight according to the latest access time of the association table; The implicit relationship reasoning unit infers potential table associations based on three-degree association path analysis.

10. A table-level data classification and grading system based on big data according to claim 9, characterized in that: In the dynamic graph calculation module, for any associated edge e between table nodes i and j, ij , its weight decays over time through an exponential decay function, the formula is: Among them, w ij (t) represents the current time t below e ij The weight of t0 represents the reference time of the last calculation or update, w ij (t0) represents the stored weight value at time t0, λ represents the time attenuation coefficient, which is used to control the attenuation speed, Indicates the timestamp when tables i and j were last accessed or updated together. That is, Δt ij , represents the time interval since the edge was last visited; Each time a table is accessed or metadata is updated, the current time t is read first and w is recalculated. ij (t), then write t into And with this w ij (t) is the new benchmark weight w ij (t0).

Images