A data processing method, apparatus, electronic device and medium

Abstract

This application provides a data processing method, apparatus, electronic device, and medium, relating to the field of data engineering technology. The method includes: determining a target field based on a first operation frequency and a second operation frequency of each field in a Data Definition Language (DDL) operation flow; calculating the path weights of a set of related paths between the target field and the first field to obtain a path weight set; calculating a comprehensive score for the first field based on the semantic similarity between the target field and the first field and the path weight set; and determining a replacement field based on the comprehensive score. The replacement field is used to replace the target field, and the replacement field includes the first field, a second field, and a third field, wherein the second field is a mixed value generated based on the first field, and the third field is a default value. This improves the flexibility and reliability of data processing and adapts to the ETL data flow requirements in scenarios with high-frequency field changes.

Description

Technical Field

[0001] This application relates to the field of data engineering technology, and in particular to a data processing method, apparatus, electronic device and medium. Background Technology

[0002] Extract-Transform-Load (ETL) is a crucial step in data flow. It is responsible for standardizing raw data scattered across multiple heterogeneous systems into high-quality data and loading it into target storage (such as data warehouses or data lakes). It is the foundation for supporting applications such as data analysis, business monitoring, and compliance reporting.

[0003] Currently, ETL solutions typically rely on developers manually writing and maintaining scripts. Data field mapping and format conversion are handled by engines with pre-defined fixed rules. When faced with schema changes, manual intervention is required to trace upstream and downstream dependencies and fix related tasks. However, manual intervention in analyzing the impact and determining remediation plans often leads to downstream data pipeline interruptions and business analysis stagnation. Furthermore, relying on human experience makes it difficult to ensure the consistency of post-processed data distribution and the continuity of business logic, resulting in data pipeline anomalies.

[0004] It is evident that existing ETL solutions suffer from poor data processing reliability in scenarios with high-frequency changes. Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, and medium that can improve the reliability of data processing in high-frequency changing scenarios.

[0006] In a first aspect, embodiments of this application provide a data processing method, the method comprising:

[0007] Based on the first and second operation frequencies of each field in the Data Definition Language (DDL) operation flow, a target field is determined. The target field is a field that meets the deletion conditions. The first operation frequency is the operation frequency of the current record of each field, and the second operation frequency is the operation frequency of the historical records of each field.

[0008] Calculate the path weights of the bloodline path set between the target field and the first field to obtain a path weight set. The path weight set includes the weights corresponding to each bloodline path in the bloodline path set. The first field is a field associated with the target field.

[0009] Based on the semantic similarity between the target field and the first field and the path weight set, the comprehensive score of the first field is calculated;

[0010] The replacement field is determined based on the comprehensive score. The replacement field is used to replace the target field. The replacement field includes the first field, the second field, and the third field. The second field is a mixed value generated based on the first field, and the third field is a default value.

[0011] Optionally, determining the target field based on the first and second operation frequencies of each field in the Data Definition Language (DDL) operation flow includes:

[0012] The deletion probability of the first changed field is determined based on the operation frequency matrix of the first changed field in the DDL operation flow within a preset time window. The first changed field is any field in the DDL operation flow that involves a change operation. The operation frequency matrix of the first changed field is determined based on the first operation frequency of the first changed field.

[0013] The deletion threshold is determined based on the first operation frequency of the first changed field and the baseline frequency of the first changed field, wherein the baseline frequency of the first changed field is determined based on the second operation frequency of the first changed field.

[0014] If the first changed field meets the deletion conditions, the first changed field is determined as the target field. The deletion conditions include: the deletion probability of the first changed field is greater than the deletion threshold, the access frequency of the first changed field is less than or equal to a preset access frequency threshold, and the first changed field does not belong to a preset field set, wherein the fields in the preset field set are used for cross-table association or data integration.

[0015] Optionally, after determining the deletion probability of the first changed field, the method further includes:

[0016] Based on the deletion probability of the first changed field and the actual detection result of the first changed field, a Brill score weighted calculation is performed to obtain the Brill score corresponding to the deletion probability of the first changed field. The actual detection result is used to indicate whether the first changed field has been deleted.

[0017] If the Brill score corresponding to the deletion probability of the first changed field is greater than or equal to a preset score threshold, the parameters of the Long Short-Term Memory (LSTM) network model are updated by gradient descent, and the LSTM model is used to determine the deletion probability of the first changed field.

[0018] Optionally, calculating the path weights of the bloodline path set between the target field and the first field to obtain the path weight set includes:

[0019] Using the target field as the starting point in the data lineage graph, the fields associated with the target field are traversed using a breadth-first search to obtain the first field;

[0020] Obtain the set of lineage paths between the target field and the first field;

[0021] The path weight of the first bloodline path is determined based on the number of times the first bloodline path is called, the sum of the number of times each bloodline path in the bloodline path set is called, and the path hop count of the first bloodline path. The first bloodline path is any bloodline path in the bloodline path set.

[0022] The path weight set is obtained based on the path weights of each bloodline path in the bloodline path set.

[0023] Optionally, calculating the comprehensive score of the first field based on the semantic similarity between the target field and the first field and the path weight set includes:

[0024] Semantic information of the target field and the first field is obtained respectively, and the semantic information includes the field name and field description;

[0025] The semantic similarity is calculated based on the cosine similarity between the semantic vector of the target field and the semantic vector of the first field, and the edit distance between the target field and the first field. The semantic vector of the target field is determined based on the semantic information of the target field, the semantic vector of the first field is determined based on the semantic information of the first field, and the edit distance is determined based on the field name of the target field and the field name of the first field.

[0026] The sum of the path weights of each bloodline path in the path weight set is calculated to obtain the aggregate weight;

[0027] The semantic similarity and the aggregation weight are weighted and summed to obtain the comprehensive score.

[0028] Optionally, determining the replacement field based on the comprehensive score includes:

[0029] If the overall score is greater than or equal to the first confidence score and the semantic similarity is greater than or equal to the preset semantic similarity threshold, the first field is determined as the replacement field.

[0030] If the overall score is less than the first confidence score and the overall score is greater than or equal to the second confidence score, the first field is linearly interpolated using a pre-trained generative adversarial network model to obtain the second field, and the second field is determined as the replacement field, wherein the second confidence score is less than the first confidence score.

[0031] If the overall score is less than the second confidence score, a third field is generated using a pre-trained generative adversarial network model according to preset constraints, and the third field is determined as the replacement field.

[0032] Optionally, after obtaining the second field or the third field, the method further includes:

[0033] Based on the data distribution of the fourth field and the historical data distribution of the target field, the Wasserstein distance between the fourth field and the target field is calculated. The fourth field is either the second field or the third field. The data distribution of the fourth field is a probability distribution obtained by fitting a batch of samples of the fourth field. The historical data distribution of the target field is a probability distribution formed by the historical true values ​​of the target field within a preset historical period.

[0034] If the Wasserstein distance is greater than or equal to a preset distance threshold, the generative adversarial network model is iterated.

[0035] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0036] The first determining module is used to determine the target field based on the first operation frequency and the second operation frequency of each field in the Data Definition Language (DDL) operation flow. The target field is a field that meets the deletion conditions. The first operation frequency is the operation frequency of the current record of each field, and the second operation frequency is the operation frequency of the historical record of each field.

[0037] The first calculation module is used to calculate the path weights of the bloodline path set between the target field and the first field to obtain a path weight set. The path weight set includes the weights corresponding to each bloodline path in the bloodline path set. The first field is a field associated with the target field.

[0038] The second calculation module is used to calculate the comprehensive score of the first field based on the semantic similarity between the target field and the first field and the path weight set;

[0039] The second determining module is used to determine a replacement field based on the comprehensive score. The replacement field is used to replace the target field. The replacement field includes the first field, the second field, and the third field. The second field is a mixed value generated based on the first field, and the third field is a default value.

[0040] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the data processing method as described in the first aspect.

[0041] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data processing method described in the first aspect.

[0042] Fifthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the data processing method as described in the first aspect.

[0043] In this embodiment, target fields meeting deletion conditions are selected based on the first and second operation frequencies of each field in the DDL operation flow, reducing monitoring of irrelevant fields and facilitating subsequent analysis and processing. Then, the path weight set and semantic similarity between the target field and the first field are calculated. The path weight set quantifies the degree of correlation between the target field and the first field in the lineage graph dimension, reducing the selection of fields with no actual data flow value. Simultaneously, semantic similarity is used for evaluation from a business semantic dimension, reducing invalid replacement fields that are closely related but semantically irrelevant. Thus, a comprehensive score is calculated based on semantic similarity and the path weight set, reducing the limitations of subjective experience-based selection or single evaluation dimensions and improving the accuracy of quantitative evaluation of the first field. Different replacement fields are determined based on differences in the comprehensive score, covering different scoring scenarios through a layered strategy, thereby reducing the occurrence of data pipeline anomalies, improving the flexibility and reliability of data processing, and adapting to ETL data flow requirements in high-frequency field change scenarios. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is one of the flowcharts of a data processing method provided in the embodiments of this application;

[0046] Figure 2 This is a second flowchart of a data processing method provided in an embodiment of this application;

[0047] Figure 3This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] See Figure 1 , Figure 1 This is a flowchart of a data processing method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0051] Step 101: Based on the first operation frequency and the second operation frequency of each field in the Data Definition Language (DDL) operation flow, determine the target field. The target field is the field that meets the deletion condition. The first operation frequency is the operation frequency of the current record of each field, and the second operation frequency is the operation frequency of the historical record of each field.

[0052] In this step, the Data Definition Language (DDL) operation flow can be monitored in real time. For example, Change Data Capture (CDC) tools (such as Debezium) can be used to capture events such as modifying table structure (ALTER TABLE) and deleting columns (DROP COLUMN). The first operation frequency can be the DDL operation frequency of each field within the current preset time window to represent the recent operation activity of the field. The second operation frequency can be the DDL operation frequency of each field within a historical preset period to represent the operation activity under normal conditions. Deletion conditions can include the probability of deleting a field being greater than a preset deletion threshold. By filtering fields that meet the deletion conditions, monitoring of irrelevant fields is reduced, facilitating subsequent analysis and processing.

[0053] For example, firstly, the first operation frequency and corresponding second operation frequency of all fields can be extracted from the DDL operation flow to construct a table-level operation frequency matrix. Then, based on the operation frequency matrix, and combined with Git commit records, ETL historical error logs, and metadata information, the deletion probability of each field is calculated using a pre-built Long Short-Term Memory (LSTM) network model. Finally, the target field is determined based on deletion conditions such as the field's deletion probability being greater than a preset deletion threshold. Using operation frequency as a feature to identify fields at risk of deletion (and target fields) provides a basis for subsequent field replacement using different strategies for the target fields.

[0054] Step 102: Calculate the path weights of the bloodline path set between the target field and the first field to obtain a path weight set. The path weight set includes the weights corresponding to each bloodline path in the bloodline path set. The first field is a field associated with the target field.

[0055] In this step, the first field can be any of the related fields obtained by traversing the data lineage graph using a breadth-first search (BFS) starting from the target field. The first field can include fields directly related to the target field, such as upstream data source fields and downstream dependent fields. It can also include fields indirectly related to the target field, such as fields that have undergone multiple jumps. The set of all lineage paths between the target field and any first field, along with the lineage path set corresponding to that first field, is used. The weight of a single lineage path can be calculated using the number of calls to the path, the sum of the call counts of all lineage paths in the lineage path set corresponding to that first field, and the number of hops. The set of weights for all lineage paths between the first field and the target field is the path weight set. The path weight set quantifies the strength of the lineage relationship between the target field and each first field. The closer the lineage relationship, the more frequent the calls, and the shorter the path, the higher the path weight, providing a basis for the lineage graph path dimension for subsequent field selection and replacement.

[0056] Step 103: Calculate the comprehensive score of the first field based on the semantic similarity between the target field and the first field and the path weight set;

[0057] In this step, the aggregated path weights are obtained by summing the path weights in the first field path weight set, which represents the strength of the lineage relationship between the target field and each of the first fields. Semantic similarity between the target field and the first fields is also calculated to represent the business semantic matching degree between them. A comprehensive score is calculated using dual-dimensional indicators to reduce the limitations of single indicators. The path weight set quantifies the closeness of the association between the target field and the first fields in terms of call frequency, path distance, and other lineage graph dimensions, reducing the selection of fields with no actual data flow value. Simultaneously, semantic similarity is used to evaluate from a business semantic dimension, reducing invalid replacement fields that are closely related but semantically unrelated. Then, a weighted fusion quantitative scoring method is adopted to provide an objective and feasible basis for determining replacement fields based on the comprehensive score, replacing subjective experience-based selection and ensuring the consistency of the processed data distribution and the coherence of the business logic.

[0058] Step 104: Determine the replacement field based on the comprehensive score. The replacement field is used to replace the target field. The replacement field includes the first field, the second field, and the third field. The second field is a mixed value generated based on the first field, and the third field is a default value.

[0059] In this step, different replacement fields can be determined based on different comprehensive scores. For example, the appropriate replacement fields can be determined according to the preset confidence scores. The replacement fields include the first field, the second field, and the third field, forming a three-level field replacement strategy to improve the stable operation of ETL data integration and cross-table join tasks after the target field is deleted.

[0060] Specifically, when the overall score reaches the high confidence score range and the semantic similarity meets the requirements, the first field, which is closely related to the target field and has a high degree of business semantic matching, is directly determined as the replacement field to achieve lossless data connection. When the overall score is in the medium confidence score range, a second field is generated based on the weighted fusion of the first field value and the value generated by the Generative Adversarial Network (GAN) model, taking into account both data relevance and business semantic integrity. When the overall score is in the low confidence score range, a third field (i.e., the default value) generated by the GAN model according to preset business constraints is used as a fallback solution to ensure that the data link is not interrupted. Finally, the layered strategy covers different scoring scenarios, thereby reducing the occurrence of anomalies in the data pipeline and improving the reliability of data processing.

[0061] In this embodiment, target fields meeting deletion conditions are selected based on the first and second operation frequencies of each field in the DDL operation flow, reducing monitoring of irrelevant fields and facilitating subsequent analysis and processing. Then, the path weight set and semantic similarity between the target field and the first field are calculated. The path weight set quantifies the degree of correlation between the target field and the first field in the lineage graph dimension, reducing the selection of fields with no actual data flow value. Simultaneously, semantic similarity is used for evaluation from a business semantic dimension, reducing invalid replacement fields that are closely related but semantically irrelevant. Thus, a comprehensive score is calculated based on semantic similarity and the path weight set, reducing the limitations of subjective experience-based selection or single evaluation dimensions and improving the accuracy of quantitative evaluation of the first field. Different replacement fields are determined based on differences in the comprehensive score, covering different scoring scenarios through a layered strategy, thereby reducing the occurrence of data pipeline anomalies, improving the flexibility and reliability of data processing, and adapting to ETL data flow requirements in high-frequency field change scenarios.

[0062] Optionally, determining the target field based on the first and second operation frequencies of each field in the Data Definition Language (DDL) operation flow includes:

[0063] The deletion probability of the first changed field is determined based on the operation frequency matrix of the first changed field in the DDL operation flow within a preset time window. The first changed field is any field in the DDL operation flow that involves a change operation. The operation frequency matrix of the first changed field is determined based on the first operation frequency of the first changed field.

[0064] The deletion threshold is determined based on the first operation frequency of the first changed field and the baseline frequency of the first changed field, wherein the baseline frequency of the first changed field is determined based on the second operation frequency of the first changed field.

[0065] If the first changed field meets the deletion conditions, the first changed field is determined as the target field. The deletion conditions include: the deletion probability of the first changed field is greater than the deletion threshold, the access frequency of the first changed field is less than or equal to a preset access frequency threshold, and the first changed field does not belong to a preset field set, wherein the fields in the preset field set are used for cross-table association or data integration.

[0066] In this embodiment, the schema change prediction module can analyze the timing characteristics of DDL operations based on an LSTM model to predict the probability of field deletion. Specifically, the first changed field can be a field from DDL change operations such as ALTER TABLE and DROP COLUMN captured by Debezium, denoted as c. iThe sliding time window can be defined as Δ = 10 minutes, and the preset time window can be [t - Δ, t]. The first change field c is statistically analyzed. i Belongs to table T i The DDL operation frequency within [t-Δ, t], i.e., the first operation frequency, is denoted as f. ij Where i is the first change field c i Belongs to table T i The index, j, is an identifier for the operation type dimension, used to distinguish different DDL operations, so f ij The frequency of changes was quantified from multiple dimensions. Then, the first operation frequency f was... ij The fine-grained temporal features are input into the LSTM model to obtain the first changed field c. i Operation frequency matrix F t F t It can be represented as:

[0067]

[0068] Among them, f ij Represented as:

[0069]

[0070] in, The value is 10 -5 .

[0071] For example, suppose a system has 3 tables (e.g., T1, T2, T3) and 2 operation types (e.g., j=1, ALTERTABLE; j=2, DROP COLUMN). Within the window [t−10min, t]: T1 executes 5 ALTER TABLE operations and 2 DROP COLUMN operations; all tables execute a total of 10 operations (i.e., the total number of operations).

[0072] At this time, f ij It can be represented as:

[0073] f 11 =5 / (10+10 −5 );

[0074] f 12 =2 / (10+10 −5 ).

[0075] In addition, the first operating frequency f ij When used as input to an LSTM model, normalization can be performed to eliminate dimensional differences and make the model more stable. The first operating frequency f after normalization is... ij It can be written as , Represented as:

[0076]

[0077] in, This represents the historical operating frequency, specifically the average of the second operating frequency. This represents the standard deviation of historical operating frequencies.

[0078] Using the above formula, the first change field c is obtained. i The operation frequency matrix F within 6 preset time windows t Next, the frequency matrix sequence {F} will be operated on. t-5Δ F t The first change field c is calculated using the LSTM model as input. i Deletion probability . It can be represented as:

[0079]

[0080] Where σ is the Sigmoid function, W o This is the weight matrix of the output layer of the LSTM model, used to calculate the field deletion probability, b o This is the bias term for the output layer of the LSTM model, used to adjust the model's output baseline and enhance its fitting ability. The LSTM model can be pre-trained using historical DDL logs, Git commit records, ETL error records, and business metadata as the training set, and employing time series cross-validation.

[0081] Considering that in high-frequency schema change scenarios, related technologies struggle to adapt to fluctuations in operation frequency (e.g., differences between peak and off-peak business periods), and that the business importance of different fields (e.g., core business fields versus log fields) varies significantly, making it difficult to differentiate the risk tolerance of critical versus non-critical fields, this example sets deletion conditions during the target field determination process, including the first changed field c. i Deletion probability Greater than the dynamic deletion threshold, first modified field c i The access frequency is less than or equal to the preset access frequency threshold, and the first change field c i This does not belong to the preset field set used for cross-table joins or data integration. The dynamic deletion threshold is based on the first changed field c. i The first operation frequency and baseline frequency are dynamically determined. In this way, by setting the deletion conditions, the selected target fields comprehensively consider multiple dimensions of indicators such as real-time monitored operation frequency, field importance, and model confidence, thus balancing risk and efficiency.

[0082] Among them, according to the first change field c i The deletion threshold is dynamically determined by the first operation frequency and the baseline frequency, as shown in the following formula:

[0083]

[0084] Threshold t This indicates a dynamically determined deletion threshold. Indicates the first changed field c i Belongs to table T i The DDL operation frequency within [t-Δ, t], i.e., the first operation frequency; The baseline frequency can be the average number of operations performed during the same period over the past 7 days. If... Much higher The threshold will be increased accordingly to avoid the risk of accidental deletion due to occasional high-frequency changes during peak business periods; conversely, the threshold will be lowered to ensure that low-frequency abnormal changes are not missed.

[0085] For example, if there are 50 DDL operations in the current 10 minutes and the baseline frequency is 30, then:

[0086] Threshold t =0.5+0.2×tanh(50 / 30)≈0.5+0.2×0.848=0.6696.

[0087] In addition, the first change field c i The access frequency is less than or equal to a preset access frequency threshold to achieve high-frequency access protection. This reduces the likelihood of fields with high-frequency access being deleted, thereby reducing data pipeline interruptions and business analysis stagnation. The first changed field is c. i The access frequency can be the first change field c i The number of visits and the first change field c i The divisor of the average daily number of visits, with a preset visit frequency threshold of 2. In other words, when the first changed field c... i The number of visits is greater than the first change field c i If the daily average number of visits is twice that of the user, then the deletion of the first change field will be prohibited.

[0088] Furthermore, the first changed field c i Not belonging to JOIN_KEYS(T) i ), JOIN_KEYS(T i ) represents the preset field set, table T iThis is the set of all fields defined as join keys. These fields are used as key identifiers (such as primary keys, foreign keys, business logic join keys, etc.) in cross-table joins (such as SQL JOIN operations) or ETL data integration. In other words, if the first changed field c is determined... i If the field belongs to the preset field set, the deletion of the first changed field is prohibited, and it is marked as high risk, generating an alarm event and notifying the operation and maintenance platform.

[0089] In one example, in the probability of deletion Greater than the dynamic deletion threshold c i The access frequency is less than or equal to 2 and c i Not belonging to JOIN_KEYS(T) i In the case of ), the first change field c i Once identified as the target field, a signal is triggered and transmitted to the lineage inference module via a Kafka message queue. The lineage inference module is used to traverse the relationships between the target field and the target field. i All related fields, get the relationship with c i The first field c, which is directly or indirectly related j .

[0090] Furthermore, in high-frequency schema change scenarios, the business impact of different fields varies significantly, and the sample size of high-frequency operation fields (such as core business tables) may be much larger than that of low-frequency fields. To improve the accuracy of field deletion probability prediction, the model prediction bias can be quantified using the Brier Score. When the bias exceeds a threshold, the LSTM parameters are updated through gradient descent, forcing the LSTM model to prioritize optimizing the prediction accuracy of high-frequency fields. This reduces the systemic risk caused by misjudgment of key fields, continuously adapts to actual business scenarios, and ensures that the calibration results more comprehensively reflect the model's true performance on key business fields, rather than being overwhelmed by a large number of low-frequency operations. See the following description for details:

[0091] In some optional embodiments, after determining the deletion probability of the first changed field, the method further includes:

[0092] Based on the deletion probability of the first changed field and the actual detection result of the first changed field, a Brill score weighted calculation is performed to obtain the Brill score corresponding to the deletion probability of the first changed field. The actual detection result is used to indicate whether the first changed field has been deleted.

[0093] If the Brill score corresponding to the deletion probability of the first changed field is greater than or equal to a preset score threshold, the parameters of the Long Short-Term Memory (LSTM) network model are updated by gradient descent, and the LSTM model is used to determine the deletion probability of the first changed field.

[0094] Among them, the deletion probability of the first changed field, i.e., the output of the LSTM model. This refers to the LSTM model's prediction of the risk of field deletion, with a value ranging from 0 to 1; the actual detection result of the first changed field. These are binary labels based on the final execution result of the DDL operation flow. For example, label 1 indicates that the first modified field underwent a delete operation such as DROPCOLUMN, while label 0 indicates that the first modified field was only modified but not deleted. (This refers to the actual detection result.) The final DDL events captured by Debezium can be used for verification, serving as a true basis for judging the accuracy of model predictions; the Brill score is used to measure the deviation between the probability prediction and the actual result, and its formula can be expressed as:

[0095]

[0096] Among them, w i The weighting coefficients can be determined by combining the number of times a field is accessed or its importance to the business, with core fields having higher weights. N is the total number of samples. Thus, according to... and By performing Brill score weighting calculation, the prediction bias weight of core fields (such as frequently accessed fields and JOIN key related fields) is increased, thus avoiding the prediction problems of core fields being masked by the random bias of ordinary fields.

[0097] If the Brillouin score obtained after weighted calculation is greater than or equal to a preset score threshold (which can be 0.05), it indicates that the deletion probability output by the LSTM model is high. Compared with the actual deletion results The deviation is too large and can no longer accurately adapt to the current DDL operation rules, so the parameters need to be updated. The updated parameters include the weight matrix W of the LSTM model output layer. o and bias term b o For example, when BS ≥ 0.05, gradient descent is used to minimize the Brill fraction as the optimization objective, and W is adjusted via backpropagation. o and b o The value of is determined by the range of values. This reduces the decline in generalization ability of the LSTM model after training, allowing the model to continuously adapt to changes in the DDL operation flow and improving the accuracy of deletion probability prediction.

[0098] The updated W o Represented as: ;

[0099] Updated b o Represented as: ;

[0100] in, For gradient, The learning rate is 0.001.

[0101] After determining the target field, the first field associated with the target field can be determined through real-time lineage graph traversal and semantic analysis. Furthermore, a comprehensive score is calculated based on semantic similarity and path weight sets to determine the replacement field. See the following description for details:

[0102] Optionally, calculating the path weights of the bloodline path set between the target field and the first field to obtain the path weight set includes:

[0103] Using the target field as the starting point in the data lineage graph, the fields associated with the target field are traversed using a breadth-first search to obtain the first field;

[0104] Obtain the set of lineage paths between the target field and the first field;

[0105] The path weight of the first bloodline path is determined based on the number of times the first bloodline path is called, the sum of the number of times each bloodline path in the bloodline path set is called, and the path hop count of the first bloodline path. The first bloodline path is any bloodline path in the bloodline path set.

[0106] The path weight set is obtained based on the path weights of each bloodline path in the bloodline path set.

[0107] In this embodiment, the real-time pedigree map and the target field c are first obtained. i Metadata, c i The metadata includes c i Table T to which it belongs i The real-time lineage graph can include data types and semantic descriptions, such as table JOIN relationships, ETL dependency paths, and field call frequencies stored in the Neo4j graph database. Then, in the Neo4j lineage graph, it is represented by c i Starting with c, traverse all pairs of c using breadth-first search. i The first field c, whether directly or indirectly related j Until the query times out (default 5 seconds), return the top-ranked first field c from the cached snapshot. j And obtain the target field c. i With any first field c j The set of all bloodline paths between them and the set of bloodline paths corresponding to the first field, each first field c jEach path corresponds to its own set of lineage paths. The weight of a single lineage path can be calculated using the path's call count, the sum of the call counts of all lineage paths in the set corresponding to the first field, and the path's hop count. The formula for calculating the weight of a single lineage path is as follows:

[0108]

[0109] in, For the first lineage path, that is, the lineage path set starting from the target field c i To any first field c j A single lineage dependency path, for example, c i →Table A→c j It is a path, c i →Table B→Table C→c j It is a separate path; express Path weights; First bloodline path The number of times it is called; For all with c i The kth path among the relevant candidate paths (k ranges from 1 to N). This represents the sum of the number of times each bloodline path in the bloodline path set is called; λ is the path hop count of the first lineage path, e is a natural constant (base approximately 2.71828) used to achieve exponential decay of the path length; λ is the path length decay factor, with a value of 0.5.

[0110] For example, if the path The number of calls was 120, the total number of calls was 1000, and the path length was... ,but, .

[0111] The set of all lineage path weights between the first field and the target field is used as the path weight set. This quantifies the strength of the lineage relationship between the target field and each of the first fields. The closer the lineage relationship, the more frequently it is called, and the shorter the path, the higher the path weight, providing a basis for the lineage path dimension in subsequent field selection and replacement.

[0112] Optionally, calculating the comprehensive score of the first field based on the semantic similarity between the target field and the first field and the path weight set includes:

[0113] Semantic information of the target field and the first field is obtained respectively, and the semantic information includes the field name and field description;

[0114] The semantic similarity is calculated based on the cosine similarity between the semantic vector of the target field and the semantic vector of the first field, and the edit distance between the target field and the first field. The semantic vector of the target field is determined based on the semantic information of the target field, the semantic vector of the first field is determined based on the semantic information of the first field, and the edit distance is determined based on the field name of the target field and the field name of the first field.

[0115] The sum of the path weights of each bloodline path in the path weight set is calculated to obtain the aggregate weight;

[0116] The semantic similarity and the aggregation weight are weighted and summed to obtain the comprehensive score.

[0117] In calculating the comprehensive score of the first field, the degree of correlation in the lineage path dimension was considered to reduce the selection of fields with no actual data flow value. Simultaneously, semantic similarity was used for evaluation from a business semantic dimension to reduce invalid substitute fields that are closely related but semantically unrelated. The process of determining the path weight set was described in the aforementioned embodiments and will not be repeated here. The process of determining the semantic similarity between the target field and the first field is described below:

[0118] First, obtain the target field c. i semantic information and the first field c j The semantic information includes field names (name) and field descriptions (desc). Field names are short text features (e.g., discount_tier), while field descriptions are long text features (e.g., order discount level, with values ​​from 0 to 5, corresponding to different discount rates). The combination of both fully reflects the business meaning of the field. For example, although the fields discount_tier and vip_discount_rate have different names, the BERT model can be used to identify the similarity of their business meanings through the descriptive text, generating highly similar vector representations.

[0119] For example, the 768-dimensional semantic vector output by the BERT model can be represented as:

[0120]

[0121] in, The field name for field c. The field description for field c; v is the semantic vector for field c. This allows us to capture the deep business semantics of the fields, improving the accuracy of subsequent calculations of semantic similarity between fields.

[0122] Then, the semantic similarity is calculated based on the cosine similarity between the semantic vector of the target field and the semantic vector of the first field, as well as the edit distance between the target field and the first field. See the following formula:

[0123]

[0124] in, For c i With c j Semantic similarity between them; c i With c j Cosine similarity between them; This is the edit distance weighting factor, with a value of 0.1; c i With c j Normalized edit distance between them; v i For c i The semantic vector, v j For c j semantic vectors, For c i field names, For c j The field names are then used. By combining cosine similarity and Levenshtein edit distance, both core business semantic matching and the intuitive similarity of field names are ensured, thus improving the accuracy of semantic similarity calculation.

[0125] For example, if c i =discount_tier, c j =vip_discount_rate, edit distance is 8, maximum length is 12, then:

[0126] The Levenshtein part = 1 − 8 / 12 = 0.3333, then Sim(c i c j =0.85 + 0.1 * 0.3333 ≈ 0.8833.

[0127] Obtain the semantic similarity between the target field and the first field. After obtaining the path weight set, the sum of the path weights of each related path in the path weight set is calculated to obtain the aggregate weight. Then, the semantic similarity and the aggregate weight are weighted and summed to obtain the comprehensive score. Specifically, the comprehensive score can be calculated using the following formula:

[0128]

[0129] in, c jThe overall score, For the path weight set, including from c i to c j All bloodline dependency paths; The aggregate weight represents the sum of the path weights of each bloodline path in the path weight set. A higher aggregate weight indicates that the data flow between the two paths is more frequent, the path is shorter, and the technical connection is closer. This is the semantic similarity weight coefficient, which can take a value of 0.6. The graph path weight coefficient can be 0.4 and can be optimized using a loss function. and The value of .

[0130] Among them, the loss function It can be represented as:

[0131]

[0132] Here, γ=0.01 is the L2 regularization coefficient to prevent overfitting. ValidAlternatives represents the set of valid alternative fields, i.e., candidate fields that have been validated as successful in historical fixes. A list of candidate replacement fields can be selected by calculating the comprehensive score. And sorted in descending order of scores.

[0133] Furthermore, the accuracy and adaptability of the cross-table lineage reasoning module can be improved by continuously learning the semantic information of newly added fields. For example, BERT can be fine-tuned hourly, inputting metadata of newly added fields, with a learning rate of η. BERT =2×10 −5 The BERT fine-tuning loss function is expressed as:

[0134]

[0135] Among them, z i For c i success rate, c i The success rate of repair directly affects its training weights. Indicates that after incremental training, c i The new semantic vector, Before incremental training, c i The old semantic vector, The mean squared error term is used to constrain the smooth transition between the old and new semantic vectors. The BERT training weights are also dynamically adjusted based on the repair success rate of each strategy in the joint repair decision. The latest JOIN frequency is calculated every 5 minutes and updated in Neo4j. value.

[0136] After obtaining a comprehensive score by weighted summation of semantic similarity and aggregation weight, the replacement field is determined based on the comprehensive score, as detailed below:

[0137] Optionally, determining the replacement field based on the comprehensive score includes:

[0138] If the overall score is greater than or equal to the first confidence score and the semantic similarity is greater than or equal to the preset semantic similarity threshold, the first field is determined as the replacement field.

[0139] If the overall score is less than the first confidence score and the overall score is greater than or equal to the second confidence score, the first field is linearly interpolated using a pre-trained generative adversarial network model to obtain the second field, and the second field is determined as the replacement field, wherein the second confidence score is less than the first confidence score.

[0140] If the overall score is less than the second confidence score, a third field is generated using a pre-trained generative adversarial network model according to preset constraints, and the third field is determined as the replacement field.

[0141] In one example, if (i.e., the overall score is greater than or equal to the first confidence score) and If the semantic similarity is greater than or equal to the preset semantic similarity threshold, the ETL SQL will be automatically modified to determine the first field as the replacement field, so as to replace c. i Replace with c j Furthermore, semantic equivalence verification (KL divergence verification) can be performed to ensure that the replaced c j The data distribution satisfies:

[0142]

[0143] If D KL If it is greater than or equal to 0.1, it indicates that c j The data distribution differs significantly from the original field, so c is automatically discarded. i Replace with c j Based on the scores of the other first fields, the replacement strategy is then applied sequentially.

[0144] In one example, if If the overall score is less than the first confidence score, but greater than or equal to the second confidence score, then a GAN model can be used to perform linear interpolation or weighted fusion on the first field to generate the second field. For example, the original value of the first field can be weighted and fused with the value generated by the GAN model that fits the business characteristics of the target field. This preserves the correlation between the first field and the target field, while the GAN-generated value compensates for semantic or data distribution mismatches, making the second field more suitable for the business and technical needs of the target field. The nonlinear interpolation parameter calculation can be based on the Sigmoid function.

[0145]

[0146] when At that time, γ = 0.5; As γ approaches 0.9, it approaches 1. The resulting mixture value... That is, the second field can be represented as:

[0147]

[0148] Where G(z) is a pre-trained GAN generator, and its output conforms to c i Composite value of historical distribution.

[0149] In one example, if (That is, if the overall score is less than the second confidence score), then it means c j With c i The semantic matching and lineage correlation of the target field are both low, offering no direct value for utilization or processing. Therefore, a fallback solution is implemented: a pre-trained GAN model is used to generate a completely new third field as a replacement field based on pre-defined business and technical constraints (such as field type, numerical range, character set, data distribution patterns, and ETL task format requirements). The core function of the pre-defined constraints is to ensure that the generated third field conforms to business logic and ETL technical specifications. Even without a suitable candidate lineage correlation field, this fallback solution can fill the gap in the target field, completely avoiding data link interruptions and ETL task errors caused by the deletion of the target field, thus achieving full-scenario data flow assurance. The third field can be represented as:

[0150]

[0151] in, The third field, which is the value generated by GAN based on preset constraints, is used to replace the deleted field c. i ;constraints represent constraints to force the generated third field to meet business rules and data distribution requirements.

[0152] Furthermore, when generating a third field (such as legacy_credit_code) using GAN, the original data may contain user privacy information (such as identity identifiers, geographic codes, etc.). Injecting Laplace noise into the generated value can ensure that attackers cannot deduce sensitive individual information from the original data using the generated data. The third field after injecting Laplace noise is represented as follows:

[0153]

[0154] Where ε represents the privacy budget, For field sensitivity, The third field after injecting Laplace noise, This is the original third field.

[0155] Finally, the repair instructions for the determined replacement fields are written to the Redis cache. Relying on Redis's high-performance read capabilities, the ETL execution engine can retrieve and use them in real time, ensuring that the replacement fields are quickly integrated into the data link and guaranteeing the continuous operation of the ETL task. If all the aforementioned replacement strategies (i.e., the generation of the first, second, and third fields) fail to be executed and no valid replacement fields can be generated, the manual review process is immediately triggered, and alarm information is generated simultaneously to notify the operation and maintenance personnel to intervene and handle the issue, so as to avoid data link interruption or ETL task abnormality due to field replacement failure.

[0156] Optionally, after obtaining the second field or the third field, the method further includes:

[0157] Based on the data distribution of the fourth field and the historical data distribution of the target field, the Wasserstein distance between the fourth field and the target field is calculated. The fourth field is either the second field or the third field. The data distribution of the fourth field is a probability distribution obtained by fitting a batch of samples of the fourth field. The historical data distribution of the target field is a probability distribution formed by the historical true values ​​of the target field within a preset historical period.

[0158] If the Wasserstein distance is greater than or equal to a preset distance threshold, the generative adversarial network model is iterated.

[0159] In this embodiment, the Wasserstein distance can be used to quantify and evaluate the consistency between the distribution of GAN-generated data and real data, and to drive the dynamic optimization of the generative model. The Wasserstein distance effectively measures the difference between non-overlapping distributions (such as sparse discrete values ​​and continuous values), avoiding evaluation failures caused by non-overlapping distributions. The Wasserstein distance can be expressed as:

[0160]

[0161] in, For historical data distribution, To generate the data distribution, specifically the data distribution for the fourth field. Represents all possible and The joint distribution set. Indicates in set One of the joint distributions selected from the data, Indicates the joint distribution The expected value is the difference between all sample pairs.

[0162] The optimization objective is: If the limit is exceeded, the GAN generator used to generate the second and third fields will be retrained.

[0163] Specifically, such as Figure 2 As shown in the figure, this application provides a data processing method to achieve automatic repair in high-frequency schema change scenarios. From change awareness and risk prediction to differentiated repair and closed-loop optimization, an automated link is formed. The logic and data flow of each module are as follows:

[0164] Schema change monitoring serves as the entry point for change awareness. The Debezium tool captures the database's DDL operation flow, such as ALTERTABLE / DROP COLUMN, in real time, ensuring that field changes are detected within seconds.

[0165] The LSTM prediction model is used for risk prediction and triggering. It can be trained on multi-source data such as historical DDL logs and Git commit records to output the field deletion probability and combine it with a dynamic deletion threshold to determine the risk level. If the deletion probability exceeds the dynamic deletion threshold, the subsequent lineage inference process is triggered.

[0166] Neo4j graph query is used for lineage relationship location. Starting from the target field, it performs a breadth-first search (BFS) in the Neo4j lineage graph, traversing downstream dependent tables, tasks and services. At the same time, it can ensure query efficiency under high concurrency through graph database load balancing, and accurately locate the affected business links.

[0167] Semantic similarity calculation and alternative field scoring are used for evaluation across both semantic and lineage dimensions. Semantic similarity calculation can be performed using an incrementally trained BERT model to calculate the cosine similarity of the semantic vectors between the target field and the first candidate field, as well as the edit distance of the field names. Alternative field scoring can be performed by fusing the aggregated weights of semantic similarity and lineage path to calculate a comprehensive score for the candidate field, providing a quantitative basis for subsequent decision-making.

[0168] The decision branch is divided into three levels based on the overall score (S), and a differentiated repair solution is matched accordingly:

[0169] When S≥0.9: SQL rewriting is performed, and the highly compatible first field is directly used as the replacement field to achieve lossless data connection; when 0.6≤S<0.9: Hybrid value generation is performed, and the first field and the GAN generated value are weighted and fused to obtain the second field, taking into account both lineage and semantic adaptation; when S<0.6: GAN default value is executed, and a third field that conforms to business constraints is generated by GAN as a fallback to avoid data link interruption.

[0170] The ETL execution engine is used to write the replacement field instructions generated by the three types of repair schemes into the cache, which is then read by the ETL execution engine in real time to complete the field replacement and business link recovery.

[0171] Furthermore, the model and its repair performance are continuously improved through real-time verification and optimization mechanisms. Wasserstein distance is used to assess the consistency of the distribution between the GAN-generated data and historical data, ensuring that the generated data conforms to business patterns. BrierScore is used to quantify the prediction bias of the LSTM model; if the bias exceeds the limit, gradient descent is used to update the model parameters, enabling online learning. Incremental training of the BERT model is employed to adapt to new field semantics. Load balancing of the Neo4j graph database ensures stability under high concurrency queries. This makes it suitable for time-sensitive business scenarios such as e-commerce promotions and financial regulation, ensuring the continuity and stability of the data chain.

[0172] The following are examples illustrating the application scenarios of this application:

[0173] Scene 1:

[0174] Before a promotional event, the online store urgently optimized its product database. The development team mistakenly deleted the `discount_tier` field from the `product_specs` table. This field was referenced by over 200 promotional rule calculation tasks, including the price calculation service (`price_calculator`) and the real-time dashboard (`flash_sale_analytics`). Debezium captured the `ALTERTABLE DROP COLUMN discount_tier` operation within 0.5 seconds and passed the event to the prediction module via a Kafka message queue. Based on historical data (this field was accessed an average of 1200 times per week), the LSTM model output the field deletion probability P_delete=0.83 and the dynamic threshold Threshold_t=0.6696. Since P_delete > Threshold_t, the automatic repair process was triggered.

[0175] In bloodline reasoning:

[0176] Graph database queries can start with `discount_tier`, perform a breadth-first search (BFS) with a depth of 4 hops, and locate the affected tasks: core services: `price_calculator` (path weight w_p=0.15) and `flash_sale_analytics` (w_p=0.12). Semantic similarity calculations can be performed using candidate fields: `vip_discount_rate` (VIP discount rate, S=0.92) and `seasonal_promotion_flag` (seasonal promotion flag, S=0.75).

[0177] In joint repair decision-making and implementation:

[0178] SQL rewriting can be done by automatically changing `WHERE discount_tier > 3` to `WHERE vip_discount_rate < 0.7`. KL divergence verification: `DKL(P(vip_discount_rate) || P(discount_tier)) = 0.07 < 0.1`, passing the verification. Hybrid value generation can be done by calculating the hybrid difference for complex promotional rules that require precise matching of historical distributions (such as "enjoy extra discounts for purchases of 3 tiers or more"). The hybrid value can be: `xhybrid = 0.5 * seasonal_promotion_flag + 0.5 * G(z)`, where `G(z)` is a synthesized value that conforms to N (μ=2.8, σ=0.5) generated by a GAN generator trained on historical `discount_tier` data.

[0179] The original logic was: CASE WHEN discount_tier > 3 THEN 'Additional Discount' ELSE 'Standard Discount' END;

[0180] The corrected logic is: CASE WHEN (0.5*seasonal_promotion_flag + 0.5*GAN_value) > 3 THEN 'Extra Discount' ELSE 'Standard Discount' END.

[0181] In real-time verification and optimization:

[0182] Distribution consistency verification can be performed using the Wasserstein distance: W(historical distribution, generated distribution) = 0.05 < 0.1, which passes the verification. Dynamic model optimization includes: Brier Score calibration: BS = 0.03 < 0.05, indicating the model's prediction accuracy meets the standard. Online learning involves incrementally updating the LSTM parameters hourly with a learning rate η = 0.001.

[0183] Effect:

[0184] From the occurrence of the DDL operation to the completion of the full-link repair, the total time was 2 minutes and 18 seconds, ensuring that the promotional activity started as planned.

[0185] Scene 2:

[0186] The core transaction system was migrated from Oracle to MySQL. The `legacy_credit_code` field in the `customer_info` table of the old system was marked as obsolete due to changes in business rules. However, this field is relied upon by the AML_Check stored procedure of the anti-money laundering system and the CBRC_Report generator for regulatory reporting. During the migration, Debezium captured the `ALTERTABLE DROP COLUMN legacy_credit_code` operation in real time, triggering a remediation process. Debezium captured the DDL operation within 0.3 seconds. The LSTM prediction module, based on historical access records (daily average calls > 800), determined that `P_delete` = 0.78, exceeding the dynamic threshold `Threshold_t` = 0.73, triggering cross-table lineage inference.

[0187] In bloodline reasoning:

[0188] A graph database query could be performed as follows: Starting with `legacy_credit_code`, a breadth-first search (BFS) is performed in the Neo4j lineage graph, traversing to a depth of 4 hops, to discover the following critical paths:

[0189] legacy_credit_code → AML_Check.procedure_input → risk_evaluation (path weight w) p =0.12);

[0190] legacy_credit_code → CBRC_Report.column_mapping → regulatory_submit (path weight w) p =0.09);

[0191] Semantic similarity analysis can be performed as follows: The semantic vector of the candidate field `risk_assessment_score` has a cosine similarity of 0.62 with the target field, but due to differences in encoding rules (including discontinued regional branch codes), the weighted score after edit distance is S=0.58, which does not reach the threshold (0.6). Other candidate fields, such as `customer_risk_level` (S=0.42), are excluded due to data type mismatch. Semantic calculation yields no direct substitute field (highest S=0.58) because the encoding rules for this field include discontinued regional branch codes, resulting in a semantic mismatch between the `risk_assessment_score` field in the new system.

[0192] In joint repair decision-making and implementation:

[0193] The GAN generator configuration can be as follows: The generator is trained based on the historical 3-year legacy credit code data (approximately 1.2 million records), with the following constraints: format: ^[AZ]{2}\d{6}$ (2 letters + 6 numbers); distribution: conforms to the historical regional distribution (60% in East China, 30% in South China, and 10% in other regions); privacy protection: inject Laplace noise (ε=0.3, sensitivity Δf=1), and generate the value x̂_i = G(z) + Laplace(0, 1 / 0.3).

[0194] In real-time verification and optimization:

[0195] Distribution consistency verification can be performed by calculating the Wasserstein distance between the generated data and the historical data, where W = 0.08 (target < 0.1), and the verification is passed.

[0196] Effect:

[0197] Within 15 minutes of the database migration being completed, the ETL engine automatically replaced the legacy_credit_code with a temporary value generated by GAN, ensuring that the anti-money laundering rules were executed without interruption and avoiding manual reconstruction of three key processes.

[0198] See Figure 3 , Figure 3 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application, as shown below. Figure 3 As shown, the data processing device 300 includes:

[0199] The first determining module 301 is used to determine a target field based on the first operation frequency and the second operation frequency of each field in the Data Definition Language (DDL) operation flow. The target field is a field that meets the deletion conditions. The first operation frequency is the operation frequency of the current record of each field, and the second operation frequency is the operation frequency of the historical record of each field.

[0200] The first calculation module 302 is used to calculate the path weights of the bloodline path set between the target field and the first field to obtain a path weight set. The path weight set includes the weights corresponding to each bloodline path in the bloodline path set. The first field is a field associated with the target field.

[0201] The second calculation module 303 is used to calculate the comprehensive score of the first field based on the semantic similarity between the target field and the first field and the path weight set;

[0202] The second determining module 304 is used to determine a replacement field based on the comprehensive score. The replacement field is used to replace the target field. The replacement field includes the first field, the second field, and the third field. The second field is a mixed value generated based on the first field, and the third field is a default value.

[0203] Optionally, the first determining module 301 is specifically used for:

[0204] The deletion probability of the first changed field is determined based on the operation frequency matrix of the first changed field in the DDL operation flow within a preset time window. The first changed field is any field in the DDL operation flow that involves a change operation. The operation frequency matrix of the first changed field is determined based on the first operation frequency of the first changed field.

[0205] The deletion threshold is determined based on the first operation frequency of the first changed field and the baseline frequency of the first changed field, wherein the baseline frequency of the first changed field is determined based on the second operation frequency of the first changed field.

[0206] If the first changed field meets the deletion conditions, the first changed field is determined as the target field. The deletion conditions include: the deletion probability of the first changed field is greater than the deletion threshold, the access frequency of the first changed field is less than or equal to a preset access frequency threshold, and the first changed field does not belong to a preset field set, wherein the fields in the preset field set are used for cross-table association or data integration.

[0207] Optionally, the first determining module 301 is further used for:

[0208] Based on the deletion probability of the first changed field and the actual detection result of the first changed field, a Brill score weighted calculation is performed to obtain the Brill score corresponding to the deletion probability of the first changed field. The actual detection result is used to indicate whether the first changed field has been deleted.

[0209] If the Brill score corresponding to the deletion probability of the first changed field is greater than or equal to a preset score threshold, the parameters of the Long Short-Term Memory (LSTM) network model are updated by gradient descent, and the LSTM model is used to determine the deletion probability of the first changed field.

[0210] Optionally, the first calculation module 302 is specifically used for:

[0211] Using the target field as the starting point in the data lineage graph, the fields associated with the target field are traversed using a breadth-first search to obtain the first field;

[0212] Obtain the set of lineage paths between the target field and the first field;

[0213] The path weight of the first bloodline path is determined based on the number of times the first bloodline path is called, the sum of the number of times each bloodline path in the bloodline path set is called, and the path hop count of the first bloodline path. The first bloodline path is any bloodline path in the bloodline path set.

[0214] The path weight set is obtained based on the path weights of each bloodline path in the bloodline path set.

[0215] Optionally, the second computing module 303 is specifically used for:

[0216] Semantic information of the target field and the first field is obtained respectively, and the semantic information includes the field name and field description;

[0217] The semantic similarity is calculated based on the cosine similarity between the semantic vector of the target field and the semantic vector of the first field, and the edit distance between the target field and the first field. The semantic vector of the target field is determined based on the semantic information of the target field, the semantic vector of the first field is determined based on the semantic information of the first field, and the edit distance is determined based on the field name of the target field and the field name of the first field.

[0218] The sum of the path weights of each bloodline path in the path weight set is calculated to obtain the aggregate weight;

[0219] The semantic similarity and the aggregation weight are weighted and summed to obtain the comprehensive score.

[0220] Optionally, the second determining module 304 is specifically used for:

[0221] If the overall score is greater than or equal to the first confidence score and the semantic similarity is greater than or equal to the preset semantic similarity threshold, the first field is determined as the replacement field.

[0222] If the overall score is less than the first confidence score and the overall score is greater than or equal to the second confidence score, the first field is linearly interpolated using a pre-trained generative adversarial network model to obtain the second field, and the second field is determined as the replacement field, wherein the second confidence score is less than the first confidence score.

[0223] If the overall score is less than the second confidence score, a third field is generated using a pre-trained generative adversarial network model according to preset constraints, and the third field is determined as the replacement field.

[0224] Optionally, the second determining module 304 is further used for:

[0225] Based on the data distribution of the fourth field and the historical data distribution of the target field, the Wasserstein distance between the fourth field and the target field is calculated. The fourth field is either the second field or the third field. The data distribution of the fourth field is a probability distribution obtained by fitting a batch of samples of the fourth field. The historical data distribution of the target field is a probability distribution formed by the historical true values ​​of the target field within a preset historical period.

[0226] If the Wasserstein distance is greater than or equal to a preset distance threshold, the generative adversarial network model is iterated.

[0227] The data processing device 300 is capable of implementing the various processes applied to the various embodiments of the data processing method described above. The technical features correspond one-to-one and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0228] This application also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described data processing method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0229] For details, see Figure 4 This application also provides an electronic device, including a bus 401, a transceiver 402, an antenna 403, a bus interface 404, a processor 405, and a memory 406.

[0230] The processor 405 is used to perform the following steps:

[0231] Based on the first and second operation frequencies of each field in the Data Definition Language (DDL) operation flow, a target field is determined. The target field is a field that meets the deletion conditions. The first operation frequency is the operation frequency of the current record of each field, and the second operation frequency is the operation frequency of the historical records of each field.

[0232] Calculate the path weights of the bloodline path set between the target field and the first field to obtain a path weight set. The path weight set includes the weights corresponding to each bloodline path in the bloodline path set. The first field is a field associated with the target field.

[0233] Based on the semantic similarity between the target field and the first field and the path weight set, the comprehensive score of the first field is calculated;

[0234] The replacement field is determined based on the comprehensive score. The replacement field is used to replace the target field. The replacement field includes the first field, the second field, and the third field. The second field is a mixed value generated based on the first field, and the third field is a default value.

[0235] exist Figure 4 In this context, a bus architecture (represented by bus 401) is used. Bus 401 can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 405 and memory represented by memory 406. Bus 401 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 404 provides an interface between bus 401 and transceiver 402. Transceiver 402 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 405 is transmitted over a wireless medium via antenna 403, which further receives data and transmits data to processor 405.

[0236] Processor 405 is responsible for managing bus 401 and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 406 can be used to store data used by processor 405 during operation.

[0237] Optionally, the processor 405 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0238] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described data processing method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0239] This application also provides a computer program product, including computer instructions. When executed by a processor, the computer instructions implement the various processes of the above-described data processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0240] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0241] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0242] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Based on the first and second operation frequencies of each field in the Data Definition Language (DDL) operation flow, a target field is determined. The target field is a field that meets the deletion conditions. The first operation frequency is the operation frequency of the current record of each field, and the second operation frequency is the operation frequency of the historical records of each field. Calculate the path weights of the bloodline path set between the target field and the first field to obtain a path weight set. The path weight set includes the weights corresponding to each bloodline path in the bloodline path set. The first field is a field associated with the target field. Based on the semantic similarity between the target field and the first field and the path weight set, the comprehensive score of the first field is calculated; The replacement field is determined based on the comprehensive score. The replacement field is used to replace the target field. The replacement field includes the first field, the second field, and the third field. The second field is a mixed value generated based on the first field, and the third field is a default value.

2. The method according to claim 1, characterized in that, The method for determining the target field based on the first and second operation frequencies of each field in the Data Definition Language (DDL) operation flow includes: The deletion probability of the first changed field is determined based on the operation frequency matrix of the first changed field in the DDL operation flow within a preset time window. The first changed field is any field in the DDL operation flow that involves a change operation. The operation frequency matrix of the first changed field is determined based on the first operation frequency of the first changed field. The deletion threshold is determined based on the first operation frequency of the first changed field and the baseline frequency of the first changed field, wherein the baseline frequency of the first changed field is determined based on the second operation frequency of the first changed field. If the first changed field meets the deletion conditions, the first changed field is determined as the target field. The deletion conditions include: the deletion probability of the first changed field is greater than the deletion threshold, the access frequency of the first changed field is less than or equal to a preset access frequency threshold, and the first changed field does not belong to a preset field set, wherein the fields in the preset field set are used for cross-table association or data integration.

3. The method according to claim 2, characterized in that, After determining the deletion probability of the first changed field, the method further includes: Based on the deletion probability of the first changed field and the actual detection result of the first changed field, a Brill score weighted calculation is performed to obtain the Brill score corresponding to the deletion probability of the first changed field. The actual detection result is used to indicate whether the first changed field has been deleted. If the Brill score corresponding to the deletion probability of the first changed field is greater than or equal to a preset score threshold, the parameters of the Long Short-Term Memory (LSTM) network model are updated by gradient descent, and the LSTM model is used to determine the deletion probability of the first changed field.

4. The method according to claim 1, characterized in that, The calculation of the path weights of the bloodline path set between the target field and the first field, to obtain the path weight set, includes: Using the target field as the starting point in the data lineage graph, the fields associated with the target field are traversed using a breadth-first search to obtain the first field; Obtain the set of lineage paths between the target field and the first field; The path weight of the first bloodline path is determined based on the number of times the first bloodline path is called, the sum of the number of times each bloodline path in the bloodline path set is called, and the path hop count of the first bloodline path. The first bloodline path is any bloodline path in the bloodline path set. The path weight set is obtained based on the path weights of each bloodline path in the bloodline path set.

5. The method according to claim 1, characterized in that, The step of calculating the comprehensive score of the first field based on the semantic similarity between the target field and the first field and the path weight set includes: Semantic information of the target field and the first field is obtained respectively, and the semantic information includes the field name and field description; The semantic similarity is calculated based on the cosine similarity between the semantic vector of the target field and the semantic vector of the first field, and the edit distance between the target field and the first field. The semantic vector of the target field is determined based on the semantic information of the target field, the semantic vector of the first field is determined based on the semantic information of the first field, and the edit distance is determined based on the field name of the target field and the field name of the first field. The sum of the path weights of each bloodline path in the path weight set is calculated to obtain the aggregate weight; The semantic similarity and the aggregation weight are weighted and summed to obtain the comprehensive score.

6. The method according to any one of claims 1 to 5, characterized in that, The process of determining the replacement field based on the comprehensive score includes: If the overall score is greater than or equal to the first confidence score and the semantic similarity is greater than or equal to the preset semantic similarity threshold, the first field is determined as the replacement field. If the overall score is less than the first confidence score and the overall score is greater than or equal to the second confidence score, the first field is linearly interpolated using a pre-trained generative adversarial network model to obtain the second field, and the second field is determined as the replacement field, wherein the second confidence score is less than the first confidence score. If the overall score is less than the second confidence score, a third field is generated using a pre-trained generative adversarial network model according to preset constraints, and the third field is determined as the replacement field.

7. The method according to claim 6, characterized in that, After obtaining the second field or the third field, the process also includes: Based on the data distribution of the fourth field and the historical data distribution of the target field, the Wasserstein distance between the fourth field and the target field is calculated. The fourth field is either the second field or the third field. The data distribution of the fourth field is a probability distribution obtained by fitting a batch of samples of the fourth field. The historical data distribution of the target field is a probability distribution formed by the historical true values ​​of the target field within a preset historical period. If the Wasserstein distance is greater than or equal to a preset distance threshold, the generative adversarial network model is iterated.

8. A data processing apparatus, characterized in that, The device includes: The first determining module is used to determine the target field based on the first operation frequency and the second operation frequency of each field in the Data Definition Language (DDL) operation flow. The target field is a field that meets the deletion conditions. The first operation frequency is the operation frequency of the current record of each field, and the second operation frequency is the operation frequency of the historical record of each field. The first calculation module is used to calculate the path weights of the bloodline path set between the target field and the first field to obtain a path weight set. The path weight set includes the weights corresponding to each bloodline path in the bloodline path set. The first field is a field associated with the target field. The second calculation module is used to calculate the comprehensive score of the first field based on the semantic similarity between the target field and the first field and the path weight set; The second determining module is used to determine a replacement field based on the comprehensive score. The replacement field is used to replace the target field. The replacement field includes the first field, the second field, and the third field. The second field is a mixed value generated based on the first field, and the third field is a default value.

9. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 7.

Images