Data confusion relation evaluation method and device, equipment and storage medium

Abstract

The invention provides a data confusion relation evaluation method and device, equipment and a storage medium, and the method comprises the steps: obtaining a creation instruction of a target table, and analyzing the table information of the target table and the field information of each field from the creation instruction; constructing a dense vector and a sparse vector of each field based on the table information and the field information of each field; and based on the dense vector and the sparse vector of each field, evaluating whether a confusion field of each field exists in a data bin in which the target table needs to be embedded, and obtaining a confusion field evaluation result of each field. By means of the method, whether the confusion relation exists between the fields in the target table and the fields existing in the data bin or not can be automatically judged, and therefore the efficiency and accuracy of data confusion relation judgment are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method, apparatus, device and storage medium for evaluating data confusion relationships. Background Art

[0002] During the design, development, and maintenance of a data warehouse, the design of data table structures and field naming standards are fundamental to ensuring data semantic clarity, ease of understanding, and usability. However, during development processes such as multi-source heterogeneous system integration, cross-team collaboration, and historical evolution, inconsistent field naming standards often lead to semantic confusion, resulting in inefficient subsequent data development and maintenance.

[0003] In existing data warehouse development and maintenance, field naming standards are often manually evaluated for consistency to determine whether field name confusion (i.e., data obfuscation) exists. This evaluation method requires significant time and effort from domain experts and developers, and due to human factors, data obfuscation is difficult to completely eliminate. Therefore, improving the efficiency and accuracy of data obfuscation assessments is a pressing technical challenge for those skilled in the art. Summary of the Invention

[0004] In view of this, the present disclosure proposes a data confusion relationship evaluation method, apparatus, device and storage medium, which can improve the efficiency and accuracy of data confusion relationship judgment.

[0005] According to a first aspect of the present disclosure, a method for evaluating data confusion relationships is provided, comprising:

[0006] Obtaining a target table creation instruction, and parsing the target table information and field information of each field from the creation instruction;

[0007] Based on the table information and the field information of each field, construct a dense vector and a sparse vector for each field;

[0008] Based on the dense vectors and sparse vectors of each of the fields, it is evaluated whether there are obfuscated fields for each of the fields in the data bin where the target table needs to be embedded, and an obfuscated field evaluation result for each of the fields is obtained.

[0009] In a possible implementation, constructing a dense vector for each field based on the table information and the field information of each field includes:

[0010] Mapping the table information into a first semantic vector;

[0011] Mapping the field information of each field into a second semantic vector of each field;

[0012] Based on the first semantic vector and the second semantic vector of each field, a dense vector of each field is constructed.

[0013] In a possible implementation, constructing a sparse vector for each field based on the table information and the field information of each field includes:

[0014] Extracting keywords from the table information to obtain keywords from the table;

[0015] Extracting keywords from the field information of each field to obtain keywords for each field;

[0016] A sparse vector of the table is constructed based on the keyword of the table and the keywords of each of the fields.

[0017] In one possible implementation, evaluating whether obfuscated fields for the fields are present in the data bin where the target table needs to be embedded based on the dense vectors and the sparse vectors of the fields, and obtaining obfuscated field evaluation results for the fields includes:

[0018] Based on the table information, a comparison table is screened out from the data warehouse;

[0019] Based on the dense vectors and the sparse vectors of each of the fields, it is evaluated whether there is a confused field for each of the fields in the comparison table, and a confused field evaluation result for each of the fields is obtained.

[0020] In a possible implementation, when evaluating whether there are obfuscated fields for each of the fields in the data bin where the target table needs to be embedded, it is implemented based on a preset obfuscation threshold.

[0021] In a possible implementation, after obtaining the obfuscated field evaluation results of the fields, the following steps are further included:

[0022] The obfuscated field evaluation results of each of the fields are parsed, and obfuscated field replacement opinions for each of the fields are generated based on the parsed results.

[0023] In a possible implementation, after generating obfuscated field replacement suggestions for each field according to the parsing results, the method further includes:

[0024] Modify the creation instruction according to the obfuscated field replacement opinions of each field;

[0025] The target table is embedded in the data store based on the modified creation instruction.

[0026] According to a second aspect of the present disclosure, a data confusion relationship evaluation device is provided, comprising:

[0027] A data acquisition module is used to acquire a target table creation instruction and parse the target table information and field information of each field from the creation instruction;

[0028] A vector construction module, configured to construct a dense vector and a sparse vector for each field based on the table information and the field information of each field;

[0029] The obfuscation evaluation module is used to evaluate whether there is an obfuscated field for each field in the data bin where the target table needs to be embedded based on the dense vector and sparse vector of each field, and obtain an obfuscated field evaluation result for each field.

[0030] According to a third aspect of the present disclosure, a data confusion relationship evaluation device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the method described in the first aspect of the present disclosure.

[0031] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the method described in the first aspect of the present disclosure.

[0032] The present disclosure provides a method, apparatus, device, and storage medium for evaluating data obfuscation relationships. The method comprises: obtaining a target table creation instruction, and parsing the target table's table information and field information of each field from the creation instruction; constructing a dense vector and a sparse vector for each field based on the table information and the field information of each field; and evaluating whether obfuscated fields for each field exist in a data warehouse where the target table is to be embedded, based on the dense vector and the sparse vector for each field, thereby obtaining an obfuscated field evaluation result for each field. The method disclosed herein can automatically determine whether a field in the target table has an obfuscation relationship with a field already existing in the data warehouse, thereby improving the efficiency and accuracy of determining data obfuscation relationships.

[0033] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0035] Figure 1 A flowchart of a method for evaluating data confusion relationships according to an embodiment of the present disclosure is shown;

[0036] Figure 2 A schematic block diagram of a data confusion relationship evaluation device according to an embodiment of the present disclosure is shown;

[0037] Figure 3 A schematic block diagram of a data confusion relationship evaluation device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0038] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0039] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0040] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0041] <Method Example>

[0042] Figure 1 FIG. 1 is a flow chart showing a method for evaluating data confusion relationships according to an embodiment of the present disclosure. Figure 1 As shown, the method includes steps S1100-S1300.

[0043] S1100, obtain the creation instruction of the target table, and parse the table information of the target table and the field information of each field from the creation instruction. Specifically, the creation instruction of the target table is used to embed the newly constructed target table in the existing data warehouse. The creation instruction includes the table information of the target table and the field information of each field. In this way, after obtaining the creation instruction of the target table, the table information of the target table and the field information of each field can be parsed from the creation instruction. The table (Table) in this application is the basic unit used to store and organize data in the data warehouse.

[0044] This table information, serving as context for each field in the table, can include: the table name, table description, and at least one of the data warehouse hierarchies in which the table is embedded. The table description records the table's background information, business implications, and technical details. The data warehouse hierarchy represents the data link dependencies of the table within the data warehouse. From top to bottom, the data warehouse hierarchy can include: the source layer (ods), the dimension layer (dim), the detail layer (dwd), the aggregation layer (dws), and the application layer (ads).

[0045] The field information of each field in the table may include at least one of: field name, field description, field data type, and field length.

[0046] After parsing the table information of the target table and the field information of each field, step S1200 may be executed to construct a dense vector and a sparse vector of each field based on the table information and the field information of each field.

[0047] In one possible implementation, constructing a dense vector for each field based on table information and field information of each field may include the following steps: mapping the table information into a first semantic vector. Mapping the field information of each field into a second semantic vector for each field. Constructing a dense vector for each field based on the first semantic vector and the second semantic vector for each field. Specifically, the dense vector for each field is obtained by concatenating the first semantic vector and the second semantic vector for each field.

[0048] For example, in an embodiment where the table information of the target table includes the table name, table description, and the data warehouse level in which the table needs to be embedded, and the field information of each field includes the field name, field description, field data type, and field length, when constructing a dense vector for one of the fields, the following steps may be included: first, the table name, table description, and the data warehouse level in which the table needs to be embedded are mapped to corresponding semantic vectors in turn, and then the semantic vectors corresponding to the table name, table description, and the data warehouse level in which the table needs to be embedded are spliced ​​to obtain a first semantic vector, that is, the first semantic vector V1 = Embedding1 (semantic vector corresponding to the data warehouse level) ⊕ Embedding2 (semantic vector corresponding to the table name) ⊕ Embedding3 (semantic vector corresponding to the table description). Next, the field name, field description, field data type, and field length of the field are mapped to corresponding semantic vectors in sequence. The semantic vectors corresponding to the field name, field description, field data type, and field length of each field are then concatenated to obtain the second semantic vector of the field: the second semantic vector V2 of the field is V2 = Embedding4 (semantic vector of the field name) ⊕ Embedding5 (semantic vector of the field description) ⊕ Embedding6 (semantic vector of the field type) ⊕ Embedding7 (semantic vector of the field length). Finally, the first semantic vector V1 and the second semantic vector V2 of the field are concatenated to obtain the dense vector V of the field. That is, the dense vector of this field V = V1⊕V2 = Embedding1 (semantic vector corresponding to the data warehouse level)⊕Embedding2 (semantic vector corresponding to the table name)⊕Embedding3 (semantic vector corresponding to the table description)⊕Embedding4 (semantic vector of the field name)⊕Embedding5 (semantic vector of the field description)⊕Embedding6 (semantic vector of the field type)⊕Embedding7 (semantic vector of the field length).

[0049] By referring to the above method, you can get the dense vector of each field in the target table.

[0050] In one possible implementation, when constructing a sparse vector for each field based on the table information and the field information of each field, the following steps may be included: extracting the keyword of the table information to obtain the keyword of the table information; extracting the keyword of the field information of each field to obtain the keyword of each field information; calculating the union of the keyword of the table information and the keyword of the field information of each field, and assigning 1 to each key in the union to obtain a sparse vector for each field.

[0051] In a possible implementation, a method for extracting keywords from table information and field information is as follows:

[0052] Keyword extraction at the data warehouse level: The data warehouse level where the direct target table is located is used as the keyword at the data warehouse level.

[0053] Keyword extraction of table names: Perform word segmentation on the table names and use the segmentation results as keywords for the table names.

[0054] Keyword extraction of table description: Perform word segmentation on the table description and use the segmentation results as keywords for the table description.

[0055] Keyword extraction of field names: perform word segmentation on the field names and use the word segmentation results as keywords for the field names.

[0056] Keyword extraction of field descriptions: perform word segmentation on the field descriptions and use the segmentation results as keywords for the field descriptions.

[0057] Keyword extraction for field types: Map field types to semantic labels, with the same type uniformly mapped (for example, CHAR, VARCHAR(32), and STRING are uniformly mapped to STRING; DATE, DATETIME, and TIMESTAMP are uniformly mapped to DATETIME). Use the semantic labels mapped to the field types as keywords for the field types. Mapping the same field type to the same semantic label can improve the probability of subsequent matching hits.

[0058] In a specific example, for the target table, the keywords of the table name, table description, data warehouse level where the table needs to be embedded, field name, field description, and field data type are extracted in sequence. The keyword extraction results are as follows:

[0059] Keyword for data warehouse level (ods): ods

[0060] Keywords for table name (customer_info): customer_info, customer, info Keywords for table description (customer information): customer information, customer, information

[0061] Keywords for the field name (cust_id): cust_id, cust, id

[0062] Keywords for field description (customer code): customer code, customer, code

[0063] Field type (VARCHAR(32) keyword: STRING

[0064] The sparse vector of the field named cust_id obtained by assigning values ​​to each keyword according to whether it appears in the field name and concatenating them is [ods=1,customer_info=1,customer=1,info=1,customer information=1,customer=1,information=1,cust_id=1,cust=1,id=1,customer code=1,code=1,STRING=1].

[0065] By referring to the above method, you can get the sparse vectors of each field in the target table.

[0066] After calculating the dense vector and sparse vector of each field, step S1300 can be executed to evaluate whether there are obfuscated fields for each field in the data warehouse where the target table needs to be embedded based on the dense vector and sparse vector of each field, and obtain the obfuscated field evaluation results of each field.

[0067] In one possible implementation, based on the dense vector and the sparse vector of each field, evaluating whether there is an obfuscated field for each field in the data bin to be embedded in the target table, and obtaining the obfuscated field evaluation result for each field may include the following steps:

[0068] First, based on the table information, the comparison table is filtered out from the data warehouse.

[0069] In a possible implementation, the data warehouse level to which the target table belongs may be extracted from the table information, and all tables in the data warehouse at the same level as or above the data warehouse level to which the target table belongs may be used as comparison tables.

[0070] In another possible implementation, in order to narrow the comparison scope, the data warehouse level, table name, and table description of the target table can be extracted from the table information, and tables with similar table names and table descriptions to the target table are preferentially extracted from tables at the same data warehouse level as comparison tables. If no similar comparison table is found in the same data warehouse level, the same method is used to screen similar comparison tables in the upper data warehouse level. And so on, until a similar comparison table is found. However, if a similar comparison table is still not retrieved after retrieving the top level of the data warehouse level (i.e., the source layer ods), all tables at the same level and above in the data warehouse to which the target table belongs are used as comparison tables.

[0071] Second, based on the dense vector and sparse vector of each field, it is evaluated whether there is a confused field for each field in the comparison table, and the confused field evaluation result of each field is obtained.

[0072] In an implementation method that uses all tables in the data warehouse at the same or higher levels as the target table as comparison tables, each field in the target table is traversed. For the current field traversed, the dense vector and sparse vector of the current field are obtained, as well as the dense vectors and sparse vectors of each field in the comparison table. Based on the dense vector and sparse vector of the current field and the dense vectors and sparse vectors of each field in the comparison table, the similarity between the current field and each field in the comparison table is calculated. Based on the similarity between the current field and each field in the comparison table, an obfuscated field evaluation result for the current field is determined. Upon completion of the traversal, the obfuscated field evaluation results for each field are obtained.

[0073] In an implementable method of using a table in the data warehouse that has a similar table name and table description to the target table as a comparison table, after obtaining the dense vector and sparse vector of the current field and the dense vector and sparse vector of each field in the comparison table, the vectors related to the table information in the dense vector and sparse vector are eliminated, and then the similarity is calculated, thereby improving the comparison efficiency. Specifically, after eliminating the vectors related to the table information, the dense vector V = V2 = Embedding4 (semantic vector of the field name) ⊕ Embedding5 (semantic vector of the field description) ⊕ Embedding6 (semantic vector of the field type) ⊕ Embedding7 (semantic vector of the field length). After eliminating the vectors related to the table information, the sparse vector is equal to the vector composed of the field information keywords and the corresponding assignment results. The specific component process is described above and will not be repeated here.

[0074] It should be noted that for each field in each table in the data warehouse, a corresponding dense vector and sparse vector are pre-stored. In this way, when performing a similarity comparison, the dense vector and sparse vector of each field in each table can be retrieved from the memory. It should be further noted that the system implementing the disclosed method will monitor the changes of each table field in the data warehouse in real time. After monitoring the changes of the table field, the dense vector and sparse vector of the table field will be recalculated, thereby ensuring the accuracy and timeliness of the dense vector and sparse vector of each table field in the data warehouse.

[0075] In one possible implementation, when calculating the similarity between the current field and field A in the comparison table based on the dense vector and sparse vector of the current field and the dense vector and sparse vector of field A in the comparison table, the following steps may be included:

[0076] First, the similarity between the dense vector of the current field and the dense vector of field A is calculated as the first similarity. The calculation formula of the first similarity is as follows:

[0077]

[0078] Where v1 is the dense vector of the current field, v2 is the dense vector of the A field, and cs is the first similarity between the dense vector of the current field and the dense vector of the A field.

[0079] Secondly, the similarity between the sparse vector of the current field and the sparse vector of field A is calculated as the second similarity. The calculation formula of the second similarity is as follows:

[0080]

[0081] Where js is the second similarity between the sparse vector of the current field and the sparse vector of field A, field1_key is the set of keywords related to the field name, field description, and field type in the sparse vector of the current field, table1_key is the set of keywords related to the table name and table description in the sparse vector of the current field, and tier1_key is the set of keywords related to the data warehouse tier in the sparse vector of the current field. field2_key is the set of keywords related to the field name, field description, and field type in the sparse vector of field A, table2_key is the set of keywords related to the table name and table description in the sparse vector of field A, and tier2_key is the set of keywords related to the data warehouse tier in the sparse vector of field A.

[0082] For example, in an embodiment where the sparse vector of the field named cust_id is [ods=1, customer_info=1, customer=1, info=1, customer information=1, customer=1, information=1, cust_id=1, cust=1, id=1, customer code=1, code=1, string=1], field1_key=[cust_id=1, cust=1, id=1, customer code=1, customer=1, code=1, string=1], table1_key=[customer_info=1, customer=1, info=1, customer information=1, customer=1, information=1], and tier1_key=[ods=1].

[0083] Finally, the calculated first similarity and second similarity are weighted and summed to obtain the similarity between the current field and field A in the comparison table. The calculation formula for the final similarity s is as follows:

[0084] s=α×cs+(1-α)×js

[0085] Here, α is the preset weight coefficient, ranging from 0 to 1. The preset weight coefficient is determined based on the degree of reliance on semantic similarity. For example, when building highly standardized data warehouse-level DWD and DWS tables, which rely more on semantic similarity than keyword matching, the weight coefficient can be set to 0.7. Data at the ODS layer may be integrated across multiple systems, resulting in relatively confusing field naming. When performing data exploration or field similarity comparisons, keyword matching is prioritized, and the weight coefficient is set to 0.3.

[0086] After calculating the similarity between the current field and field A in the comparison table, the obfuscation field evaluation result of the current field can be determined based on the calculated similarity. When evaluating whether the obfuscation field of each field exists in the data warehouse to be embedded in the target table, it is implemented based on a pre-set obfuscation threshold.

[0087] Specifically, when 0 ≤ s ≤ m, the current field is deemed to have a low similarity with Field A in the comparison table, indicating no possibility of confusion. Therefore, a confusion field assessment result is output, indicating no confusion exists between the current field and Field A in the comparison table. When m < s < 1, the current field is deemed to have a high similarity or consistency with Field A in the comparison table, indicating a possibility of confusion. Therefore, a confusion field assessment result is output, indicating a possible confusion exists between the current field and Field A in the comparison table. When s = 1, the calculation logic associated with the current field and Field A in the comparison table is obtained. This calculation logic is recorded in the field information of each field. After obtaining the calculation logic associated with the current field and Field A in the comparison table, the consistency of the calculation logic is compared. If the calculation logic is consistent, a confusion field assessment result is output, indicating no confusion exists between the current field and Field A in the comparison table. If the calculation logic is inconsistent, a confusion field assessment result is output, indicating a confusion exists between the current field and Field A in the comparison table, prompting the customer to verify the information. m is the confusion threshold, which can be set as needed. For example, m can be set to 0.5.

[0088] In one possible implementation, in order to improve the flexibility and accuracy of the evaluation, the obfuscation threshold m can be dynamically adjusted according to the level of the data warehouse, the business importance of the field, and the historical obfuscation frequency.

[0089] First, dynamically adjust the obfuscation threshold m based on the data warehouse hierarchy. Specifically, the semantic consistency and standardization requirements for data vary across different data warehouse hierarchies. Therefore, different obfuscation thresholds can be set for different hierarchies.

[0090] For the source layer (ods): Since the data comes directly from the business system, the naming conventions may be confusing. Therefore, the confusion threshold can be appropriately lowered to increase sensitivity. For example, when the target table and the comparison table are from the source layer, when evaluating whether the fields in the target table are confused with the fields in the comparison table, the default confusion threshold m can be adjusted from 0.5 to 0.4 to capture more potentially confusing fields.

[0091] For the aggregation layer (DWS / ADS): Because the data at this level is cleansed and processed and is highly standardized, the confusion threshold can be appropriately increased to reduce false positives. For example, when evaluating whether fields in the target table are confused with fields in the comparison table, the default confusion threshold m can be adjusted from 0.5 to 0.6, thereby only issuing warnings for highly similar fields.

[0092] Second, dynamically adjust the confusion threshold m based on the business importance of the field. Specifically, naming consistency of core business fields has a greater impact on the system, while naming consistency of non-critical fields has a smaller impact. Therefore, the confusion threshold m can be dynamically adjusted based on the importance of the field. The importance of business fields can be pre-set based on user needs and stored as field information.

[0093] For high-importance fields (such as user ID and transaction amount), the confusion threshold can be increased to ensure that warnings are only issued for highly similar fields. For example, if the current field is a high-importance field, the confusion threshold can be increased from 0.5 to 0.7 to avoid frequent false positives that interfere with core fields.

[0094] For low-importance fields (such as notes and temporary fields), you can lower the obfuscation threshold to quickly detect possible obfuscation issues. For example, you can lower the obfuscation threshold from 0.5 to 0.3 to improve detection coverage.

[0095] Third, the confusion threshold m is dynamically adjusted according to the historical confusion frequency.

[0096] Specifically, for frequently obfuscated fields, the obfuscation threshold can be appropriately lowered. For example, if a pair of fields (such as "Customer ID" and "User Code") is marked as obfuscated multiple times, its obfuscation threshold will be automatically lowered to enhance detection sensitivity. For example, if a pair of fields is manually confirmed as obfuscated three times, the obfuscation threshold will be automatically adjusted from 0.5 to 0.4.

[0097] For infrequently obfuscated fields, the obfuscation threshold can be appropriately increased. For example, if a field hasn't been obfuscated for a long time, the threshold can be appropriately increased to reduce redundant prompts. For example, if the "Creation Time" field has never been obfuscated in the history, the obfuscation threshold can be automatically increased from 0.5 to 0.6.

[0098] By dynamically adjusting the confusion threshold, the system can adapt to different data scenarios more intelligently, reduce manual intervention, and improve the accuracy of confusion detection.

[0099] In one possible implementation, to enhance the depth and accuracy of field confusion assessment, a field lineage analysis mechanism can be introduced. This mechanism helps determine potential confusion risks between fields by tracking upstream and downstream dependencies within the data processing chain. Specifically, this may include the following steps:

[0100] First, a field lineage map is constructed. Specifically, the system records the complete processing path for each field, including the original source field (e.g., source layer field), intermediate processing steps (e.g., ETL transformation logic), and derived target fields (e.g., aggregation layer fields). For example, if the "Order Amount" field is rounded off from the "Transaction Amount" field in the source layer, the system will establish a lineage relationship between the two fields and record the specific transformation logic.

[0101] Second, implement cross-level confusion detection. Specifically, when calculating the similarity between the current field and field A in the comparison table, the blood relationship between the two fields will also be checked: if there is a direct blood relationship (such as a parent-child field relationship), even if the similarity s does not reach the threshold, it will be marked as "potential confusion"; if there is an indirect blood relationship (such as having a common upstream field), the similarity judgment standard will be appropriately lowered. For example, when evaluating the "member points" field, although the similarity with the "customer points" field is only 0.45 (lower than the standard 0.5), because both are derived from the "user points" field, the system will still prompt "the two fields may be related, it is recommended to check".

[0102] Third, conduct an impact analysis. Specifically, once a confusion relationship is confirmed, the system automatically traces all downstream tables and reports affected by the confused field, generates a detailed impact report, and provides batch correction suggestions. For example, if confusion between "User ID" and "Customer Number" is confirmed, the system will list all 10 reports and 5 data dashboards using these two fields and recommend revising them to "User ID."

[0103] This implementation utilizes lineage analysis to uncover underlying confusion that traditional semantic matching might miss, providing a more comprehensive perspective on field associations and significantly reducing the risk of misjudgment due to a lack of understanding of data processing history. For example, within a bank data warehouse, lineage analysis can reveal that the "customer number" in the loan system (derived from the core system) and the "user number" in the deposit system (derived from the same core system)—despite significant differences in field names and descriptions—are correctly identified as the same entity due to their lineage association, thus avoiding subsequent data association errors.

[0104] In one possible implementation, after obtaining the obfuscation field evaluation results for each field, the method further includes parsing the obfuscation field evaluation results for each field and generating obfuscation field replacement suggestions for each field based on the parsed results. Specifically, if the obfuscation field evaluation results indicate that there may be an obfuscation relationship between the current field and field A in the comparison table, field A in the comparison table is used as the replacement suggestion for the current field.

[0105] In one possible implementation, after generating obfuscated field replacement suggestions for each field based on the parsing results, the method further includes: modifying the creation instructions based on the obfuscated field replacement suggestions for each field; and embedding the target table in the data warehouse based on the modified creation instructions. This method can modify the creation instructions of the target table, thereby achieving automatic modification of the creation instructions and automatic embedding of the target table.

[0106] The following example further illustrates the above method. The specific steps are as follows: When building a table other than ODS, the target table's creation statement is parsed to extract metadata (including the data warehouse level, table name, table description, field names, field descriptions, and field types) for the target table. Dense and sparse vectors are then constructed. Confusion relationships are used to evaluate the similarity between the target table fields and the fields in the dense and sparse vector base database. The matching principle prioritizes matching dense and sparse vectors with the same data warehouse level and table name keywords. If similarity does not meet the requirements, similarity is then evaluated against dense and sparse vectors from the previous data warehouse level until a match is found at the ODS level. If a base database field that meets the similarity requirements is found, recommendations are provided for target table field information to improve field consistency. For example, it is recommended to change the target table fields (cust_no, customer number, ord_date, order time) to the base database fields (customer_id, customer code, order_date, order date). If no highly similar fields are found, recommendations are not provided and the target table can be generated automatically. In addition, this method supports similarity monitoring of existing table fields, provides modification suggestions, realizes rapid normalization of field information, and corrects the basic library of dense vectors and sparse vectors.

[0107] If the data in the data warehouse ODS source layer comes from the database environment of multiple business systems, and the metadata of the source layer data is consistent with the original data, the inconsistent field information between different systems will bring difficulties to the cross-system multi-table data association analysis. The system provides data exploration capabilities to analyze the similarity between the target master table and the fields of other system data tables in the ODS layer, and map the fields with high similarity (for example, the "order_no" field in the A master table and the "transaction_id" field in the B table are highly similar, both are order numbers, and the A and B tables can be associated and analyzed through similar fields). Through the standard matching module evaluation of cross-system data in the source layer, the compatibility and consistency of cross-system data integration are guaranteed, while improving query efficiency and ensuring data quality.

[0108] The present disclosure provides a method for evaluating data confusion relationships, including: obtaining a target table creation instruction, and parsing the target table's table information and field information of each field from the creation instruction; constructing dense vectors and sparse vectors of each field based on the table information and field information of each field; based on the dense vectors and sparse vectors of each field, evaluating whether there are confusion fields for each field in the data warehouse where the target table needs to be embedded, and obtaining confusion field evaluation results for each field. The method disclosed herein can automatically determine whether a field in the target table has a confusion relationship with a field already existing in the data warehouse, thereby improving the efficiency and accuracy of data confusion relationship judgment. Labor costs can be reduced by 70% and processing speed increased by 40%.

[0109] Furthermore, the method disclosed herein can assist developers in constructing table field information, avoid the problem of inconsistent field information due to personal reasons, avoid field confusion in subsequent data use, and improve development efficiency.

[0110] Furthermore, the present disclosure combines semantic models with keyword matching to more accurately identify field obfuscation, increasing the accuracy from the traditional 72% to 90%.

[0111] Furthermore, the disclosed method can support metadata exploration and obfuscated field assessment across multiple business systems, improving the analyzability of inter-system data. Cross-system field matching, such as correctly associating order_no in an order system with transaction_id in a payment system, increased the accuracy rate from 55% to 80%.

[0112] <Device Example>

[0113] Figure 2 FIG. 1 is a schematic block diagram of a data confusion relationship evaluation device according to an embodiment of the present disclosure. Figure 2 As shown, the device 100 includes:

[0114] The data acquisition module 110 is used to acquire a target table creation instruction and parse the target table information and field information of each field from the creation instruction;

[0115] A vector construction module 120, configured to construct a dense vector and a sparse vector for each field based on the table information and the field information of each field;

[0116] The obfuscation evaluation module 130 is used to evaluate whether there is an obfuscated field for each field in the data warehouse to be embedded in the target table based on the dense vector and sparse vector of each field, and obtain the obfuscation field evaluation result of each field.

[0117] <Equipment Example>

[0118] Figure 3 FIG. 1 is a schematic block diagram of a data confusion relationship evaluation device according to an embodiment of the present disclosure. Figure 3 As shown, the data confusion relationship evaluation device 200 includes: a processor 210 and a memory 220 for storing executable instructions of the processor 210. The processor 210 is configured to implement any of the above-mentioned data confusion relationship evaluation methods when executing the executable instructions.

[0119] It should be noted that there may be one or more processors 210. Furthermore, the data obfuscation relationship assessment device 200 according to the present embodiment may further include an input device 230 and an output device 240. The processor 210, memory 220, input device 230, and output device 240 may be connected via a bus or other means, which are not specifically limited herein.

[0120] Memory 220, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and various modules, such as the programs or modules corresponding to the data obfuscation relationship evaluation method according to the present disclosure. Processor 210 executes the software programs or modules stored in memory 220 to perform various functional applications and data processing of data obfuscation relationship evaluation device 200.

[0121] The input device 230 may be used to receive input numbers or signals. The signals may be key signals related to user settings and function control of the device / terminal / server. The output device 240 may include a display device such as a display screen.

[0122] <Storage Medium Embodiment>

[0123] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium is further provided, on which computer program instructions are stored. When the computer program instructions are executed by the processor 210, any of the above-mentioned data confusion relationship evaluation methods is implemented.

[0124] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technical improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for evaluating data confusion relationships, characterized in that: include: Obtaining a target table creation instruction, and parsing the target table information and field information of each field from the creation instruction; Based on the table information and the field information of each field, construct a dense vector and a sparse vector for each field; Based on the dense vectors and sparse vectors of each of the fields, it is evaluated whether there are obfuscated fields for each of the fields in the data bin where the target table needs to be embedded, and an obfuscated field evaluation result for each of the fields is obtained.

2. The method according to claim 1, characterized in that When constructing a dense vector for each field based on the table information and the field information of each field, the method includes: Mapping the table information into a first semantic vector; Mapping the field information of each field into a second semantic vector of each field; Based on the first semantic vector and the second semantic vector of each field, a dense vector of each field is constructed.

3. The method according to claim 1, characterized in that When constructing a sparse vector for each field based on the table information and the field information of each field, the method includes: Extracting keywords from the table information to obtain keywords from the table; Extracting keywords from the field information of each field to obtain keywords for each field; A sparse vector of the table is constructed based on the keyword of the table and the keywords of each of the fields.

4. The method according to claim 1, wherein When evaluating whether there is an obfuscated field for each field in the data bin where the target table needs to be embedded based on the dense vector and the sparse vector of each field, and obtaining an obfuscated field evaluation result for each field, the method includes: Based on the table information, a comparison table is screened out from the data warehouse; Based on the dense vectors and the sparse vectors of each of the fields, it is evaluated whether there is a confused field for each of the fields in the comparison table, and a confused field evaluation result for each of the fields is obtained.

5. The method according to claim 1, wherein When evaluating whether there are obfuscated fields for each of the fields in the data bin to be embedded in the target table, this is achieved based on a preset obfuscation threshold.

6. The method according to claim 1, characterized in that After obtaining the obfuscation field evaluation results of each field, it also includes: The obfuscated field evaluation results of each of the fields are parsed, and obfuscated field replacement opinions for each of the fields are generated based on the parsed results.

7. The method according to claim 6, characterized in that After generating the obfuscated field replacement opinions for each field according to the parsing results, it also includes: Modify the creation instruction according to the obfuscated field replacement opinions of each field; The target table is embedded in the data store based on the modified creation instruction.

8. A data confusion relationship evaluation device, characterized in that: include: A data acquisition module is used to acquire a target table creation instruction and parse the target table information and field information of each field from the creation instruction; A vector construction module, configured to construct a dense vector and a sparse vector for each field based on the table information and the field information of each field; The obfuscation evaluation module is used to evaluate whether there is an obfuscated field for each field in the data bin where the target table needs to be embedded based on the dense vector and sparse vector of each field, and obtain an obfuscated field evaluation result for each field.

9. A data confusion relationship evaluation device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method according to any one of claims 1 to 7 when executing the executable instructions.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images