Data obfuscation relationship assessment methods, apparatus, equipment and storage media

Abstract

This disclosure provides a method, apparatus, device, and storage medium for evaluating data obfuscation relationships. The method includes: obtaining a creation instruction for a target table, and parsing table information and field information of each field from the creation instruction; constructing dense vectors and sparse vectors for each field based on the table information and field information; and evaluating whether obfuscated fields exist in the data warehouse to which the target table needs to be embedded, based on the dense and sparse vectors, to obtain the obfuscation field evaluation results for each field. The method of this disclosure can automatically determine whether fields in the target table have obfuscation relationships with existing fields in the data warehouse, thereby improving the efficiency and accuracy of data obfuscation relationship determination.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a method, apparatus, device and storage medium for evaluating data confusion relationships. Background Technology

[0002] In the design, development, and maintenance of a data warehouse, the design of the data table structure and the standardization of field naming are fundamental to ensuring that the data is semantically clear, easy to understand, and easy to use. However, in the development process involving the integration of multi-source heterogeneous systems, cross-team collaboration, and historical evolution, semantic confusion often arises due to inconsistent field naming standards, leading to low efficiency in subsequent data development and maintenance.

[0003] In the current development and maintenance of data warehouses, consistency assessment of field naming conventions is typically performed manually to determine the existence of field name confusion issues (i.e., data confusion problems). This assessment method requires a significant investment of time and effort from domain experts and developers, and due to human factors, data confusion problems are difficult to completely eliminate. Therefore, improving the efficiency and accuracy of data confusion assessment is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, this disclosure proposes a method, apparatus, device and storage medium for evaluating data confusion relationships, which can improve the efficiency and accuracy of data confusion relationship judgment.

[0005] According to a first aspect of this disclosure, a method for evaluating data obfuscation relationships is provided, comprising:

[0006] Obtain the creation instruction of the target table, and parse the table information and field information of each field of the target table from the creation instruction;

[0007] Based on the table information and the field information of each field, construct the dense vector and sparse vector of each field;

[0008] Based on the dense and sparse vectors of each field, the presence of obfuscated fields for each field in the data warehouse into which the target table needs to be embedded is evaluated, and the obfuscated field evaluation results for each field are obtained.

[0009] In one possible implementation, when constructing the dense vector of each of the fields based on the table information and the field information of each of the fields, the process includes:

[0010] Map the table information to a first semantic vector;

[0011] Map the field information of each field to a second semantic vector of each field;

[0012] Based on the first semantic vector and the second semantic vector of each field, a dense vector of each field is constructed.

[0013] In one possible implementation, when constructing the sparse vector of each field based on the table information and the field information of each field, the process includes:

[0014] Extract the keywords from the table information to obtain the table's keywords;

[0015] Extract the keywords from the field information of each field to obtain the keywords of each field;

[0016] Based on the key of the table and the key of each field, construct a sparse vector of the table.

[0017] In one possible implementation, when evaluating whether obfuscated fields exist in the data warehouse to which the target table needs to be embedded, based on the dense and sparse vectors of each of the fields, and obtaining the obfuscated field evaluation results for each of the fields, the process includes:

[0018] Based on the information in the table, a comparison table is selected from the data warehouse;

[0019] Based on the dense and sparse vectors of each field, the existence of obfuscated fields for each field in the comparison table is evaluated to obtain the obfuscated field evaluation results for each field.

[0020] In one possible implementation, when assessing whether there are obfuscated fields for each of the fields in the data warehouse into which the target table needs to be embedded, the assessment is based on a pre-set obfuscation threshold.

[0021] In one possible implementation, after obtaining the obfuscation field evaluation results for each of the aforementioned fields, the method further includes:

[0022] The obfuscated field evaluation results of each field are parsed, and obfuscated field replacement suggestions are generated based on the parsing results.

[0023] In one possible implementation, after generating obfuscated field replacement suggestions for each of the fields based on the parsing results, the method further includes:

[0024] Based on the proposed replacement of obfuscated fields for each of the aforementioned fields, modify the creation instructions;

[0025] The target table is embedded in the data warehouse based on the modified creation instructions.

[0026] According to a second aspect of this disclosure, a data obfuscation relationship assessment apparatus is provided, comprising:

[0027] The data acquisition module is used to acquire the creation instruction of the target table, and parse the table information and field information of each field of the target table from the creation instruction;

[0028] A vector construction module is used to construct dense vectors and sparse vectors for each of the fields based on the table information and the field information of each field.

[0029] The obfuscation evaluation module is used to evaluate whether obfuscated fields exist in the data warehouse into which the target table needs to be embedded, based on the dense and sparse vectors of each field, and to obtain the obfuscation field evaluation result of each field.

[0030] According to a third aspect of this disclosure, a data obfuscation relationship evaluation apparatus is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the method described in the first aspect of this disclosure.

[0031] According to a fourth aspect of this disclosure, a non-volatile computer-readable storage medium is provided that stores computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement the method described in the first aspect of this disclosure.

[0032] This disclosure provides a method, apparatus, device, and storage medium for evaluating data obfuscation relationships. The method includes: obtaining a creation instruction for a target table, and parsing table information and field information of each field from the creation instruction; constructing dense vectors and sparse vectors for each field based on the table information and field information; and evaluating whether obfuscated fields exist in the data warehouse to which the target table needs to be embedded, based on the dense and sparse vectors, to obtain the obfuscation field evaluation results for each field. The method of this disclosure can automatically determine whether fields in the target table have obfuscation relationships with existing fields in the data warehouse, thereby improving the efficiency and accuracy of data obfuscation relationship determination.

[0033] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0034] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0035] Figure 1 A flowchart illustrating a data obfuscation relationship evaluation method according to an embodiment of the present disclosure is shown;

[0036] Figure 2 A schematic block diagram of a data obfuscation relationship evaluation apparatus according to an embodiment of the present disclosure is shown;

[0037] Figure 3 A schematic block diagram of a data obfuscation relationship evaluation apparatus according to an embodiment of the present disclosure is shown. Detailed Implementation

[0038] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0039] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0040] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0041] <Method Implementation>

[0042] Figure 1 A flowchart illustrating a data obfuscation relationship evaluation method according to an embodiment of this disclosure is shown. Figure 1 As shown, the method includes steps S1100-S1300.

[0043] S1100: Obtain the creation instruction for the target table, and parse the table information and field information of each field from the creation instruction. Specifically, the creation instruction for the target table is used to embed the newly constructed target table into an existing data warehouse. This creation instruction includes the table information and field information of each field. Thus, after obtaining the creation instruction for the target table, the table information and field information of each field can be parsed from it. In this application, a table is the basic unit used to store and organize data in a data warehouse.

[0044] The information in this table, serving as contextual information for each field, can include: table name, table description, and at least one of the data warehouse layers into which the table needs to be embedded. The table description records the table's background information, business implications, and technical details. The data warehouse layer represents the data link dependencies of a table within the data warehouse. The data warehouse layer can be arranged from top to bottom as follows: ODS (Original Data Storage), DM (Dimensional Data Storage), DWD (Detail Data Storage), DWS (Aggregation Data Storage), and ADS (Application Data Storage).

[0045] The field information for each field in the table may include at least one of the following: field name, field description, field data type, and field length.

[0046] After parsing the table information and field information of the target table, step S1200 can be executed to construct dense vectors and sparse vectors for each field based on the table information and field information.

[0047] In one possible implementation, constructing the dense vector of each field based on table information and field information can include the following steps: mapping the table information to a first semantic vector; mapping the field information of each field to a second semantic vector of each field; and constructing the dense vector of each field based on the first semantic vector and the second semantic vector of each field. Specifically, the dense vector of each field can be obtained by concatenating the first semantic vector and the second semantic vector of each field.

[0048] For example, in an embodiment where the target table's information includes the table name, table description, and the data warehouse level into which the table needs to be embedded, and the field information for each field includes the field name, field description, field data type, and field length, the following steps can be included when constructing a dense vector for one of the fields: First, the table name, table description, and the data warehouse level into which the table needs to be embedded are mapped sequentially to corresponding semantic vectors. Then, the semantic vectors corresponding to the table name, table description, and the data warehouse level into which the table needs to be embedded are concatenated to obtain the first semantic vector, i.e., the first semantic vector V1 = Embedding1 (semantic vector corresponding to the data warehouse level) ⊕ Embedding2 (semantic vector corresponding to the table name) ⊕ Embedding3 (semantic vector corresponding to the table description). Next, the field name, field description, field data type, and field length are mapped sequentially to corresponding semantic vectors. Then, the semantic vectors corresponding to the field name, field description, field data type, and field length are concatenated to obtain the second semantic vector of the field. That is, the second semantic vector of the field V2 = Embedding4 (semantic vector of field name) ⊕ Embedding5 (semantic vector of field description) ⊕ Embedding6 (semantic vector of field type) ⊕ Embedding7 (semantic vector of field length). Finally, the first semantic vector V1 and the second semantic vector V2 of the field are concatenated to obtain the dense vector V of the field. That is, the dense vector of the field V = V1⊕V2 = Embedding1 (semantic vector corresponding to the data warehouse level)⊕Embedding2 (semantic vector corresponding to the table name)⊕Embedding3 (semantic vector corresponding to the table description)⊕Embedding4 (semantic vector of the field name)⊕Embedding5 (semantic vector of the field description)⊕Embedding6 (semantic vector of the field type)⊕Embedding7 (semantic vector of the field length).

[0049] By following the method described above, the dense vectors of each field in the target table can be obtained.

[0050] In one possible implementation, when constructing the sparse vector of each field based on table information and field information of each field, the following steps may be included: extracting the keywords of the table information to obtain the keywords of the table information; extracting the keywords of the field information of each field to obtain the keywords of the field information; calculating the union of the keywords of the table information and the keywords of the field information of each field, and assigning the value of each key in the union to 1, thereby obtaining the sparse vector of each field.

[0051] In one possible implementation, the method for extracting keywords from table information and field information is as follows:

[0052] Keyword extraction at the data warehouse level: The data warehouse level where the target table resides is used as the key at the data warehouse level.

[0053] Keyword extraction from table names: The table names are segmented into words, and the segmentation results are used as the keywords for the table names.

[0054] Keyword extraction from table descriptions: The table descriptions are segmented into words, and the segmentation results are used as keywords for the table descriptions.

[0055] Keyword extraction from field names: The field names are segmented into words, and the segmentation results are used as keywords for the field names.

[0056] Keyword extraction from field descriptions: The field descriptions are segmented into words, and the segmentation results are used as keywords for the field descriptions.

[0057] Keyword extraction for field types: Field types are mapped to semantic tags, with the same type being mapped uniformly (e.g., CHAR, VARCHAR(32), and STRING are uniformly mapped to STRING; DATE, DATETIME, and TIMESTAMP are uniformly mapped to DATETIME). The semantic tags mapped to field types are used as keywords for the field types. Mapping field types of the same type to the same semantic tags can improve the probability of subsequent matching hits.

[0058] In a specific example, for the target table, keywords are extracted sequentially: table name, table description, data warehouse level to be embedded, field name, field description, and field data type. The keyword extraction results are shown below:

[0059] Keyword for data warehouse level (ODS): ODS

[0060] Keyword for table name (customer_info): customer_info, customer, info Keyword for table description (customer information): customer information, customer, information

[0061] Keywords for field name (cust_id): cust_id, cust, id

[0062] Keywords for the field description (customer code): customer code, customer, code

[0063] The keyword for the field type (VARCHAR(32)) is STRING.

[0064] Based on whether each keyword appears in the field name, assign values to them and splice them. The sparse vector of the field named cust_id is [ods=1, customer_info=1, customer=1, info=1, customer information=1, customer=1, information=1, cust_id=1, cust=1, id=1, customer code=1, code=1, STRING=1].

[0065] Referring to the above method, the sparse vectors of each field in the target table can be obtained.

[0066] After calculating the dense vectors and sparse vectors of each field, step S1300 can be executed. Based on the dense vectors and sparse vectors of each field, evaluate whether there are any confusing fields of each field in the data warehouse that the target table needs to be embedded into, and obtain the evaluation results of the confusing fields of each field.

[0067] In a possible implementation manner, when evaluating whether there are any confusing fields of each field in the data warehouse that the target table needs to be embedded into, based on the dense vectors and sparse vectors of each field, and obtaining the evaluation results of the confusing fields of each field, the following steps may be included:

[0068] First, based on the table information, select the comparison tables from the data warehouse.

[0069] In a possible implementation manner, the data warehouse level to which the target table belongs can be extracted from the table information, and all tables in the data warehouse at the same level and above as the data warehouse level to which the target table belongs are used as comparison tables.

[0070] In another possible implementation manner, in order to narrow the comparison range, the data warehouse level, table name, and table description of the target table can be extracted from the table information. First, extract the tables similar to the table name and table description of the target table from the tables at the same data warehouse level as comparison tables. If no similar comparison tables are found at the same data warehouse level, the same method is used to screen for similar comparison tables in the data warehouse level above. And so on, until similar comparison tables are found. However, if no similar comparison tables are retrieved even when reaching the top level of the data warehouse level (i.e., the source layer ods), then all tables in the data warehouse at the same level and above as the data warehouse level to which the target table belongs are used as comparison tables.

[0071] Second, based on the dense vectors and sparse vectors of each field, evaluate whether there are any confusing fields of each field in the comparison tables, and obtain the evaluation results of the confusing fields of each field.

[0072] In one possible approach of using tables at all levels within the data warehouse that are at the same or higher level as the target table as the comparison table, the process involves traversing each field in the target table. For the current field, the dense and sparse vectors of that field, along with the dense and sparse vectors of each field in the comparison table, are obtained. Based on these vectors, the similarity between the current field and each field in the comparison table is calculated. Finally, based on this similarity, the obfuscation evaluation result for the current field is determined. Once the traversal is complete, the obfuscation evaluation results for each field are obtained.

[0073] In the feasible approach of using tables in the data warehouse with similar names and descriptions to the target table as comparison tables, after obtaining the dense and sparse vectors of the current field and the dense and sparse vectors of each field in the comparison table, vectors related to table information are removed from the dense and sparse vectors before similarity calculation is performed, thereby improving comparison efficiency. Specifically, after removing vectors related to table information, the dense vector V = V2 = Embedding4 (semantic vector of field name) ⊕ Embedding5 (semantic vector of field description) ⊕ Embedding6 (semantic vector of field type) ⊕ Embedding7 (semantic vector of field length). After removing vectors related to table information, the sparse vector is equal to the vector composed of field information keywords and their corresponding assignment results. See the above for the specific component process, which will not be repeated here.

[0074] It's important to note that each field in each table of the data warehouse has its corresponding dense and sparse vectors pre-stored. This allows for the retrieval of these vectors from memory during similarity comparisons. Furthermore, the system executing this method monitors changes to the fields in each table of the data warehouse in real time. Upon detecting a change, it recalculates the dense and sparse vectors for that field, ensuring the accuracy and timeliness of the dense and sparse vectors for each table field in the data warehouse.

[0075] In one possible implementation, calculating the similarity between the current field and field A in the comparison table based on the dense and sparse vectors of the current field and the dense and sparse vectors of field A in the comparison table may include the following steps:

[0076] First, the similarity between the dense vector of the current field and the dense vector of field A is calculated as the first similarity. The formula for calculating the first similarity is as follows:

[0077]

[0078] In the formula, v1 is the dense vector of the current field, v2 is the dense vector of field A, and cs is the first similarity between the dense vector of the current field and the dense vector of field A.

[0079] Secondly, the similarity between the sparse vector of the current field and the sparse vector of field A is calculated as the second similarity. The formula for calculating the second similarity is as follows:

[0080]

[0081] In the formula, js represents the second similarity between the sparse vector of the current field and the sparse vector of field A; field1_key represents the set of keywords related to field name, field description, and field type in the sparse vector of the current field; table1_key represents the set of keywords related to table name and table description in the sparse vector of the current field; tier1_key represents the set of keywords related to data warehouse level in the sparse vector of the current field; field2_key represents the set of keywords related to field name, field description, and field type in the sparse vector of field A; table2_key represents the set of keywords related to table name and table description in the sparse vector of field A; and tier2_key represents the set of keywords related to data warehouse level in the sparse vector of field A.

[0082] For example, in an embodiment where the sparse vector of the field named cust_id is [ods=1,customer_info=1,customer=1,info=1,customer information=1,customer=1,information=1,cust_id=1,cust=1,id=1,customer code=1,code=1,STRING=1], field1_key=[cust_id=1,cust=1,id=1,customer code=1,customer=1,code=1,STRING=1], table1_key=[customer_info=1,customer=1,info=1,customer information=1,customer=1,information=1], tier1_key=[ods=1].

[0083] Finally, the calculated first and second similarities are weighted and summed to obtain the similarity between the current field and field A in the comparison table. The formula for calculating the final similarity s is shown below:

[0084] s = α × cs + (1 - α) × js

[0085] In the formula, α is a preset weighting coefficient, with a value between 0 and 1. The preset weighting coefficient is determined based on the degree of reliance on semantic similarity. For example, when constructing highly normalized data warehouse hierarchical DWD and DWS tables, semantic similarity is more important than keyword matching, so the weighting coefficient can be set to 0.7. Data in the ODS layer may be integrated from multiple systems, and field naming may be relatively chaotic. When conducting data exploration or field similarity comparison, keyword matching is prioritized, so the weighting coefficient for this layer is set to 0.3.

[0086] After calculating the similarity between the current field and field A in the comparison table, the evaluation result of the obfuscated field for the current field can be determined based on the calculated similarity. When evaluating whether obfuscated fields exist in the data warehouse to be embedded in the target table, this is done based on a pre-set obfuscation threshold.

[0087] Specifically, when 0 ≤ s ≤ m, the current field is considered to have low similarity to field A in the comparison table, and there is no possibility of confusion. Therefore, an evaluation result indicating that the current field and field A in the comparison table are not confused is output. When m < s < 1, the current field is considered to have high similarity to field A in the comparison table, or they are identical, and there is a possibility of confusion. Therefore, an evaluation result indicating that the current field and field A in the comparison table may be confused is output. When s = 1, the calculation logic related to the current field and field A in the comparison table is obtained. This calculation logic is recorded in the field information of each field. After obtaining the calculation logic related to the current field and field A in the comparison table, the consistency of the calculation logic is compared. If the calculation logic is consistent, an evaluation result indicating that the current field and field A in the comparison table are not confused is output. If the calculation logic is inconsistent, an evaluation result indicating that the current field and field A in the comparison table are confused is output, prompting the customer to verify. Here, m is the confusion threshold, which can be set according to requirements. For example, m can be set to 0.5.

[0088] In one possible implementation, to improve the flexibility and accuracy of the assessment, the obfuscation threshold m can be dynamically adjusted based on the data warehouse hierarchy, the business importance of the fields, and the historical obfuscation frequency.

[0089] First, the obfuscation threshold m is dynamically adjusted based on the data warehouse hierarchy. Specifically, the semantic consistency and standardization requirements of data differ across different levels of the data warehouse. Therefore, different obfuscation thresholds can be set for different levels.

[0090] For the source layer (ODS): Since the data comes directly from the business system, the naming conventions may be inconsistent. Therefore, the obfuscation threshold can be appropriately lowered to improve sensitivity. For example, when the target table and the comparison table both originate from the source layer, the default obfuscation threshold m can be adjusted from 0.5 to 0.4 when evaluating whether fields in the target table are obfuscated with fields in the comparison table, so that more potentially obfuscated fields can be captured.

[0091] For the aggregation layer (DWS / ADS): Since the data at this level has been cleaned and processed, its standardization is relatively high. Therefore, the obfuscation threshold can be appropriately increased to reduce false alarms. For example, when the target table and the comparison table come from the aggregation layer, when evaluating whether fields in the target table are obfuscated with fields in the comparison table, the default obfuscation threshold m can be adjusted from 0.5 to 0.6, so that warnings are only issued for highly similar fields.

[0092] Second, the obfuscation threshold *m* is dynamically adjusted based on the business importance of the fields. Specifically, naming consistency of core business fields has a significant impact on the system, while naming consistency of non-critical fields has a smaller impact. Therefore, the obfuscation threshold *m* can be dynamically adjusted based on the importance of the fields. The importance of business fields can be pre-defined according to user needs and stored as field information.

[0093] For highly important fields (such as user ID and transaction amount), the obfuscation threshold can be increased to ensure that warnings are only issued for highly similar fields. For example, when the current field is highly important, the obfuscation threshold can be increased from 0.5 to 0.7 to avoid frequent false alarms that may interfere with core fields.

[0094] For low-importance fields (such as notes and temporary fields), the obfuscation threshold can be lowered to quickly detect potential obfuscation issues. For example, the obfuscation threshold can be lowered from 0.5 to 0.3 to improve detection coverage.

[0095] Third, the confusion threshold m is dynamically adjusted based on the historical confusion frequency.

[0096] Specifically, for frequently obfuscated fields, the obfuscation threshold can be appropriately lowered. For example, if a pair of fields (such as "Customer ID" and "User Code") is marked as obfuscated multiple times, its obfuscation threshold will be automatically lowered to enhance detection sensitivity. For example, if a pair of fields is manually confirmed as obfuscated three times, the obfuscation threshold will be automatically adjusted from 0.5 to 0.4.

[0097] For low-frequency obfuscated fields, the obfuscation threshold can be appropriately increased. For example, if a field has not been marked for obfuscation for a long time, the threshold can be appropriately increased to reduce redundant prompts. For example, if the field "creation time" has never been marked for obfuscation in the history, the obfuscation threshold will be automatically increased from 0.5 to 0.6.

[0098] By dynamically adjusting the obfuscation threshold, the system can more intelligently adapt to different data scenarios, reduce manual intervention, and improve the accuracy of obfuscation detection.

[0099] In one possible implementation, to enhance the depth and accuracy of obfuscation field assessment, a field lineage analysis mechanism can be introduced. This mechanism helps determine the potential obfuscation risk between fields by tracing the upstream and downstream dependencies of fields in the data processing chain. Specifically, it may include the following steps:

[0100] First, the system constructs a field lineage graph. Specifically, it records the complete processing path of each field, including: the original source field (such as the source layer field), the intermediate processing steps (such as ETL transformation logic), and the derived target field (such as the aggregation layer field). For example, when it is found that the "order amount" field is calculated by rounding the "total transaction amount" field in the source layer, the system establishes a lineage relationship between the two fields and records the specific transformation logic.

[0101] Second, cross-level confusion detection is implemented. Specifically, when calculating the similarity between the current field and field A in the comparison table, the lineage relationship between the two fields is also checked: if there is a direct lineage relationship (such as a parent-child field relationship), it will be marked as "potential confusion" even if the similarity s does not reach the threshold; if there is an indirect lineage relationship (such as having a common upstream field), the similarity judgment standard will be appropriately lowered. For example, when evaluating the "Member Points" field, although the similarity with the "Customer Points" field is only 0.45 (below the standard of 0.5), because both originate from the "User Points" field, the system will still prompt "These two fields may be related, and it is recommended to check."

[0102] Third, conduct obfuscation impact analysis. Specifically, when obfuscation is confirmed, the system will automatically trace all downstream tables and reports affected by the obfuscated fields, generate a detailed impact scope report, and provide batch correction suggestions. For example, when it is confirmed that "User ID" and "Customer Number" are obfuscated, the system will list all 10 reports and 5 data dashboards that use these two fields and suggest uniformly changing them to "User ID".

[0103] In this feasible approach, by introducing lineage analysis, deep-seated confusion issues that traditional semantic matching may miss can be discovered, providing a more comprehensive perspective on field relationships and significantly reducing the risk of misjudgment due to a lack of understanding of data processing history. For example, in a bank's data warehouse, lineage analysis reveals that the "customer number" in the loan system (originating from the core system) and the "user number" in the deposit system (originating from the same core system), although with significant differences in field names and descriptions, are correctly identified as the same entity due to their lineage relationship, avoiding subsequent data association errors.

[0104] In one possible implementation, after obtaining the obfuscation field evaluation results for each field, the method further includes: parsing the obfuscation field evaluation results for each field, and generating obfuscation field replacement opinions for each field based on the parsing results. Specifically, when the obfuscation field evaluation result indicates that the current field may have an obfuscation relationship with field A in the comparison table, field A in the comparison table is used as the replacement opinion for the current field.

[0105] In one possible implementation, after generating obfuscated field replacement suggestions for each field based on the parsing results, the method further includes: modifying the creation instruction based on the obfuscated field replacement suggestions; and embedding the target table into the data warehouse based on the modified creation instruction. This method allows for the modification of the target table's creation instruction, thereby achieving automatic modification of the creation instruction and automatic embedding of the target table.

[0106] The following example further illustrates the above method. The specific steps are as follows: When constructing tables other than the ODS (Operations Data Storage) layer, the table creation statement of the target table is parsed to extract metadata information (including data warehouse level, table name, table description, field names, field descriptions, and field types), constructing dense and sparse vectors. Obfuscation relationships are used to evaluate the similarity between the target table fields and the base library fields of the dense and sparse vectors. The matching principle prioritizes matching dense and sparse vectors with the same data warehouse level and table name keywords. If the similarity requirement is not met, it is then compared with the dense and sparse vectors of the previous data warehouse level until a match is found at the data warehouse level (ODS layer). If a base library field that meets the similarity requirement is matched, reference suggestions are provided for the target table field information to improve the consistency of the field information. For example, it is suggested to change the target table fields `cust_no` (customer ID) and `ord_date` (order time) to the base library similarity vector fields `customer_id` (customer code) and `order_date` (order date). If no highly similar fields are found, no reference suggestions are provided, and the target table can be constructed automatically. In addition, this method supports similarity monitoring of existing table fields, provides modification suggestions, enables rapid standardization of field information, and corrects the dense vector and sparse vector base library.

[0107] If the data in the ODS (Operational Data Storage) source layer comes from database environments of multiple business systems, and the metadata of the source layer data is consistent with the original data, inconsistencies in field information between different systems can cause difficulties in cross-system multi-table data association analysis. The system provides data exploration capabilities to analyze the similarity of fields between the target master table and other system data tables in the ODS layer. Fields with high similarity are mapped (e.g., the "order_no" field in master table A and the "transaction_id" field in table B are highly similar, both representing order numbers; similar fields can be used to perform association analysis between tables A and B). The system uses a standard matching module for cross-system data in the source layer for evaluation to ensure compatibility and consistency of cross-system data integration, while improving query efficiency and ensuring data quality.

[0108] This disclosure provides a method for evaluating data obfuscation relationships, including: obtaining the creation instruction of the target table, and parsing the table information and field information of each field from the creation instruction; constructing dense vectors and sparse vectors for each field based on the table information and field information; and evaluating whether obfuscated fields exist in the data warehouse to which the target table needs to be embedded, based on the dense and sparse vectors of each field, to obtain the obfuscation field evaluation results for each field. This method can automatically determine whether fields in the target table have obfuscation relationships with existing fields in the data warehouse, thereby improving the efficiency and accuracy of data obfuscation relationship judgment. Manual costs can be reduced by 70%, and processing speed can be increased by 40%.

[0109] Furthermore, the method disclosed herein can assist developers in constructing table field information, avoiding inconsistencies in field information due to personal reasons, preventing confusion of fields used in subsequent data, and improving development efficiency.

[0110] Furthermore, this disclosure achieves more accurate identification of field obfuscation through the combination of semantic modeling and keyword matching. Accuracy is improved from the traditional 72% to 90%.

[0111] Furthermore, the method disclosed herein can support metadata exploration and obfuscated field evaluation across multiple business systems, improving the analyzability of data between systems. The correct association rate between the order system's `order_no` and the payment system's `transaction_id` across systems has increased from 55% to 80%.

[0112] <Device Embodiment>

[0113] Figure 2 A schematic block diagram of a data obfuscation relationship evaluation apparatus according to an embodiment of the present disclosure is shown. Figure 2 As shown, the device 100 includes:

[0114] The data acquisition module 110 is used to acquire the creation instruction of the target table, and parse the table information and field information of each field of the target table from the creation instruction;

[0115] The vector construction module 120 is used to construct dense vectors and sparse vectors for each field based on the table information and the field information of each field.

[0116] The obfuscation evaluation module 130 is used to evaluate whether obfuscated fields exist in the data warehouse to be embedded in the target table based on the dense and sparse vectors of each field, and to obtain the obfuscation field evaluation result for each field.

[0117] <Equipment Example>

[0118] Figure 3 A schematic block diagram of a data obfuscation relationship evaluation apparatus according to an embodiment of the present disclosure is shown. Figure 3 As shown, the data obfuscation relationship evaluation device 200 includes a processor 210 and a memory 220 for storing executable instructions of the processor 210. The processor 210 is configured to implement any of the data obfuscation relationship evaluation methods described above when executing the executable instructions.

[0119] It should be noted here that the number of processors 210 can be one or more. Furthermore, the data obfuscation relationship evaluation device 200 of this embodiment may also include an input device 230 and an output device 240. The processors 210, memory 220, input device 230, and output device 240 can be connected via a bus or other means, without specific limitations here.

[0120] The memory 220, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and various modules, such as the program or module corresponding to the data obfuscation relationship evaluation method of this disclosure embodiment. The processor 210 executes various functional applications and data processing of the data obfuscation relationship evaluation device 200 by running the software program or module stored in the memory 220.

[0121] Input device 230 can be used to receive input digital numbers or signals. These signals may include key signals related to user settings and function control of the device / terminal / server. Output device 240 may include a display device such as a screen.

[0122] <Storage Medium Examples>

[0123] According to a fourth aspect of this disclosure, a non-volatile computer-readable storage medium is also provided, on which computer program instructions are stored, which, when executed by processor 210, implement any of the data obfuscation relationship evaluation methods described above.

[0124] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method of data obfuscation relationship evaluation, the method comprising: The method comprises the following steps: obtaining the creation instruction of the target table, and parsing the table information of the target table and the field information of each field from the creation instruction, wherein the table information includes at least one of the table name, the table description and the data warehouse level that needs to be embedded in the table as the context information of each field in the table, and the field information of each field in the table includes at least one of the field name, the field description, the field data type and the field length; based on the table information and the field information of each field, constructing the dense vector and the sparse vector of each field; based on the dense vector and the sparse vector of each field, evaluating whether there is a confused field of each field in the data warehouse that needs to be embedded in the target table, and obtaining the confused field evaluation result of each field; when constructing the dense vector of each field based on the table information and the field information of each field, the method comprises the following steps: mapping the table name, the table description and the data warehouse level that needs to be embedded in the table into corresponding semantic vectors in sequence, and then splicing the corresponding semantic vectors of the table name, the table description and the data warehouse level that needs to be embedded in the table to obtain a first semantic vector; mapping the field name, the field description, the field data type and the field length of each field into corresponding semantic vectors in sequence, and then splicing the corresponding semantic vectors of the field name, the field description, the field data type and the field length of each field to obtain a second semantic vector of each field; splicing the first semantic vector and the second semantic vector of each field to construct the dense vector of each field; when constructing the sparse vector of each field based on the table information and the field information of each field, the method comprises the following steps: extracting the key of the table information to obtain the key of the table information; extracting the key of the field information of each field to obtain the key of each field information; calculating the union of the key of the table information and the key of the field information of each field, and assigning 1 to each key in the union to obtain the sparse vector of each field.

2. The method of claim 1, wherein, when evaluating whether there is a confused field of each field in the data warehouse that needs to be embedded in the target table based on the dense vector and the sparse vector of each field, and obtaining the confused field evaluation result of each field, the method comprises the following steps: based on the table information, screening a comparison table from the data warehouse; based on the dense vector and the sparse vector of each field, evaluating whether there is a confused field of each field in the comparison table, and obtaining the confused field evaluation result of each field.

3. The method of claim 1, wherein, When evaluating whether there is a confused field of each field in the data warehouse that needs to be embedded in the target table, the evaluation is realized based on a pre-set confusion threshold.

4. The method of claim 1, wherein, After obtaining the confused field evaluation result of each field, the method further comprises the following steps: parsing the confused field evaluation result of each field, and generating a confused field replacement opinion of each field according to the parsing result.

5. The method of claim 4, wherein, After generating the confused field replacement opinion of each field according to the parsing result, the method further comprises the following steps: modifying the creation instruction according to the confused field replacement opinion of each field; embedding the target table in the data warehouse based on the modified creation instruction.

6. A data obfuscation relationship evaluation apparatus characterized by comprising: The method comprises the following steps: The data acquisition module is configured to acquire a creation instruction of a target table, and parse table information of the target table and field information of each field from the creation instruction, wherein the table information, as context information of each field in the table, comprises at least one of a table name, a table description, and a data warehouse level to be embedded by the table, and the field information of each field in the table comprises at least one of a field name, a field description, a field data type, and a field length; The vector construction module is configured to construct dense vectors and sparse vectors of each field based on the table information and the field information of each field; The confusion evaluation module is configured to evaluate whether there is a confusion field of each field in a data warehouse to be embedded by the target table based on the dense vectors and the sparse vectors of each field, and obtain a confusion field evaluation result of each field. In the construction of the dense vectors of each field based on the table information and the field information of each field, the following steps are included: The table name, the table description, and the data warehouse level to be embedded by the table are sequentially mapped into corresponding semantic vectors, and then the semantic vectors corresponding to the table name, the table description, and the data warehouse level to be embedded by the table are spliced to obtain a first semantic vector; The field name, the field description, the field data type, and the field length of each field are sequentially mapped into corresponding semantic vectors, and then the semantic vectors corresponding to the field name, the field description, the field data type, and the field length of each field are spliced to obtain a second semantic vector of each field; The first semantic vector and the second semantic vectors of each field are spliced to construct the dense vectors of each field. In the construction of the sparse vectors of each field based on the table information and the field information of each field, the following steps are included: extracting a keyword of the table information to obtain the keyword of the table information; extracting a keyword of the field information of each field to obtain the keyword of each field information; calculating a union set of the keyword of the table information and the keyword of the field information of each field, and assigning 1 to each keyword in the union set to obtain the sparse vectors of each field.

7. A data obfuscation relationship evaluation device characterized by comprising: The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 5. The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 5. The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 5. ​ 8. A non-transitory computer readable storage medium having stored thereon computer program instructions, wherein, ​

Images