Field similarity analysis method, table similarity analysis method and storage medium

Abstract

The invention discloses a field similarity analysis method, a table similarity analysis method and a storage medium, and relates to the technical field of data processing.The method comprises the steps that a first field and a second field with the similarity to be analyzed are obtained, a pre-constructed processing link weight graph is obtained, the processing link weight graph comprises field nodes and weight connection edges, and the field nodes are connected with the weight connection edges; field nodes connected with the weight connection edges have an operation processing relation, and the edge weights of the weight connection edges are set based on the operation processing relation; identifying a processing path from the first field to the second field in the processing link weight map; the field similarity between the first field and the second field is determined according to the processing path, and the field similarity is in positive correlation with the edge weight of the weight connection edge in the processing path. According to the method and the device, efficient and high-accuracy field similarity analysis is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a field similarity analysis method, a table similarity analysis method and a storage medium. BACKGROUND

[0002] In a financial data warehouse environment, the number of data tables is large, the structure is complex, and the sources are diverse, which brings significant challenges to data governance and consistency guarantee. Especially in the business table processing process, there may be multiple tables with repeated processing caliber in the same business field; and between different business fields, there are often similar or even identical processing logic in public tables (such as customer tables, transaction tables, etc.). Such redundancy not only leads to waste of storage resources, but also may cause data management confusion, consistency problems and repeated development of subsequent ETL processes. Therefore, effective similarity analysis of data tables to identify redundant tables and repeated processing logic has become a key requirement to improve data governance efficiency.

[0003] Traditional field similarity analysis methods can be mainly divided into three categories: structural similarity, content similarity and semantic similarity. However, the above methods all have the problem of being difficult to guarantee the efficiency and accuracy of the calculation at the same time. The structural similarity method is fast in calculation speed, but its accuracy is easily greatly reduced due to problems such as non-standard field naming, synonym diversity or abbreviation diversity. The content similarity method, such as the evaluation based on cosine similarity, faces the problem of sharp rise in computational complexity when dealing with high-dimensional fields, leading to too long analysis time and difficulty in meeting the needs of financial data warehouse for rapid comparison of large-scale data tables. The semantic similarity method can improve the accuracy of understanding, but it relies on complex NLP (Natural Language Processing) models or knowledge graphs, which consume huge computing resources, and the model training and inference process is cumbersome and inefficient.

[0004] Therefore, how to realize efficient and high-accuracy field similarity analysis is a technical problem to be solved at present. SUMMARY

[0005] The main purpose of the present application is to provide a field similarity analysis method, a table similarity analysis method and a storage medium, aiming at solving the technical problem of how to realize efficient and high-accuracy field similarity analysis.

[0006] To achieve the above purpose, the present application provides a field similarity analysis method, which comprises:

[0007] obtaining a first field and a second field for which similarity is to be analyzed, and obtaining a pre-constructed processing link weight graph, wherein the processing link weight graph comprises field nodes and weight connection edges, there is an operation processing relationship between the field nodes connected by the weight connection edges, and the edge weight of the weight connection edge is set based on the operation processing relationship;

[0008] identifying a processing path from the first field to the second field in the processing link weight graph;

[0009] determining the field similarity between the first field and the second field according to the processing path, wherein the field similarity is positively correlated with the edge weight of the weight connection edge in the processing path.

[0010] In an embodiment, the processing path includes at least one unbranched processing path from the first field to the second field, and the step of determining the field similarity between the first field and the second field according to the processing path comprises:

[0011] multiplying the edge weights of the weight connection edges in each unbranched processing path to obtain a single-path similarity, wherein the edge weight of each weight connection edge is greater than or equal to zero or less than or equal to one;

[0012] calculating the average of all single-path similarities to obtain the field similarity between the first field and the second field.

[0013] In an embodiment, before the step of obtaining a pre-constructed processing link weight graph, the method further comprises:

[0014] obtaining a pre-constructed processing link logic graph, wherein the processing link logic graph comprises field nodes and weightless connection edges, and there is an operation processing relationship between the field nodes connected by the weightless connection edges;

[0015] setting the edge weight of each weightless connection edge in the processing link logic graph according to the operation processing relationship corresponding to each weightless connection edge;

[0016] After the edge weight of each weightless connection edge in the processing link logic graph is set, the pre-constructed processing link weight graph is obtained.

[0017] In an embodiment, the step of setting the edge weight of each weight connection edge in the processing link logic graph according to the operation processing relationship corresponding to each weight connection edge comprises:

[0018] traversing each weightless connection edge in the processing link logic graph to obtain all operation processing relationships corresponding to the weightless connection edge;

[0019] obtaining an operation weight corresponding to each operation processing relationship according to a preset mapping relationship, wherein the preset mapping relationship is a corresponding relationship between different operation processing relationships and operation weights;

[0020] multiplying all the operation weights to obtain an edge weight of the edge without weight.

[0021] In an embodiment, before the step of obtaining the pre-constructed processing link logical graph, the method further comprises:

[0022] converting each SQL statement in the data processing job script into a minimum processing unit SDP;

[0023] constructing a SQL-level directed acyclic graph according to a syntax operation relationship between the minimum processing units SDP;

[0024] analyzing a table dependency relationship between each SQL-level directed acyclic graph, and constructing a job-level directed acyclic graph based on the table dependency relationship;

[0025] constructing a link-level directed acyclic graph according to a job dependency relationship corresponding to the job-level directed acyclic graph;

[0026] performing a data operation model operation on the link-level directed acyclic graph to generate an optimized link-level directed acyclic graph;

[0027] determining that the optimized link-level directed acyclic graph is the pre-constructed processing link logical graph.

[0028] In an embodiment, the step of identifying a processing path from the first field to the second field in the processing link weight graph comprises:

[0029] if the first field and a downstream field in the second field are multi-job processing fields, sequentially traversing each job processing window;

[0030] identifying a processing path from the first field to the second field based on the processing link weight graph under the job processing window;

[0031] The step of determining a field similarity between the first field and the second field according to the processing path comprises:

[0032] determining a field similarity between the first field and the second field under each job processing window according to the processing path under each job processing window, so as to obtain an average value of all the field similarities after obtaining the field similarities under all the job processing windows to obtain a global similarity between the first field and the second field.

[0033] Further, to achieve the above object, the present application provides a table similarity analysis method, comprising the following steps:

[0034] obtaining a first data table and a second data table to be analyzed for similarity;

[0035] calculating field similarity between any two fields in the first data table and the second data table, wherein the field similarity is determined by the field similarity analysis method according to any one of claims 1 to 5;

[0036] determining table similarity between the first data table and the second data table according to each field similarity, wherein the table similarity is positively correlated with each field similarity.

[0037] In an embodiment, the step of determining table similarity between the first data table and the second data table according to each field similarity comprises:

[0038] constructing an initial similarity matrix, wherein the number of rows of the initial similarity matrix is the total number of fields of the first data table, and the number of columns of the similarity matrix is the total number of fields of the second data table;

[0039] writing each field similarity into the corresponding element position in the initial similarity matrix, and writing a preset initial value into the remaining element positions to obtain a target similarity matrix, wherein the similarity between the i-th field in the first data table and the j-th field in the second data is corresponding to the element position of the i-th row and the j-th column of the initial similarity matrix;

[0040] traversing each row element value of the target similarity matrix to calculate the average value of all target element values in each row element value, wherein the target element value is an element value that is not the preset initial value;

[0041] dividing the sum value between all the average values by the number of rows of the target similarity matrix to obtain the table similarity between the first data table and the second data.

[0042] In an embodiment, the step of calculating field similarity between any two fields in the first data table and the second data table, the method further comprises:

[0043] if the downstream data table in the first data table and the second data table is a multi-job processing table, then each job processing window is traversed in turn;

[0044] calculating field similarity between any two fields in the first data table and the second data table under the job processing window;

[0045] The step of determining the table similarity between the first data table and the second data table according to the similarity of each field comprises:

[0046] The table similarity between the first data table and the second data under each job processing window is determined according to the similarity of each field under each job processing window, respectively, so as to obtain the global similarity between the first data table and the second data table by calculating the average value of all the table similarities after obtaining all the table similarities under all the job processing windows.

[0047] In addition, to achieve the above object, the present application also provides a readable storage medium, which is a computer readable storage medium, and a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to implement the steps of the method as described above.

[0048] The present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method as described above.

[0049] The one or more technical solutions proposed in the present application have at least the following technical effects:

[0050] The embodiment of the present application effectively overcomes the inherent defects of traditional structure similarity, content similarity and semantic similarity methods in terms of efficiency and accuracy by introducing a pre-constructed processing link weight graph. Specifically, after obtaining the first field and the second field to be analyzed for similarity, the embodiment of the present application identifies the processing path existing between the fields in the pre-constructed processing link weight graph, and quantifies the similarity according to the edge weight set based on the operation processing relationship in the path, effectively avoiding the misjudgment caused by the surface differences such as non-standard naming, synonyms or abbreviations of the traditional structure similarity method, thereby significantly improving the accuracy and interpretability of the results. In terms of computational efficiency, there is no need to perform complex content calculation in a high-dimensional vector space (such as cosine similarity), and the dependence on large NLP models or knowledge graphs with huge computational resource consumption is completely avoided. By converting the problem into a path search and weight aggregation problem in the pre-constructed graph structure, the efficiency of the graph structure algorithm can be fully utilized, and the computational complexity is greatly reduced, so that the actual demand for large-scale and batch field comparison of massive data tables in the financial data warehouse can be quickly responded. In summary, the field similarity analysis method of the embodiment of the present application successfully realizes the coordinated improvement of analysis efficiency and result accuracy by skillfully utilizing the processing blood relationship of data itself, i.e. the operation processing relationship, and provides a reliable technical foundation for application scenarios such as data governance, impact analysis and blood tracing. BRIEF DESCRIPTION OF DRAWINGS

[0051] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application.

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings are only some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative work should be within the protection scope of the present application.

[0053] Figure 1 A flowchart of a first embodiment of the field similarity analysis method of the present application;

[0054] Figure 2 A schematic diagram of a processing link weight graph involved in an embodiment of the field similarity analysis method of the present application;

[0055] Figure 3 A processing path schematic diagram involved in an embodiment of the field similarity analysis method of the present application;

[0056] Figure 4 An SDP schematic diagram involved in an embodiment of the field similarity analysis method of the present application;

[0057] Figure 5 A processing link logic diagram schematic diagram involved in an embodiment of the field similarity analysis method of the present application;

[0058] Figure 6 A Figure 5 A processing link weight graph schematic diagram;

[0059] Figure 7 A device structure schematic diagram of a hardware running environment involved in the field similarity analysis method device in the embodiments of the present application.

[0060] The purposes, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0061] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative work should be within the protection scope of the present application.

[0062] The financial data warehouse contains a large number of data tables, which are complex in structure and diverse in source, bringing great difficulties to data governance and consistency guarantee. In the same business field, there may be repeated processing caliber for a certain business table. Even in different fields, some public tables (such as transaction tables, customer tables, etc.) may also have similar or identical processing logic. Therefore, in order to identify redundant tables or duplicate data and avoid storage waste and management confusion caused by data redundancy, similar model analysis is a difficult problem that must be faced.

[0063] Traditional similarity measurement methods include structural similarity, content similarity and semantic similarity. Structural similarity needs to compare field names, types, constraints, etc., but non-standard field naming or synonym problems may affect accuracy; content similarity needs to analyze field data distribution (such as statistical characteristics, correlation), but similarity calculation of high-dimensional data (such as cosine similarity) is very time-consuming; semantic similarity needs to combine natural language processing (NLP) or knowledge graph technology to understand the business meaning of the field, which has a high technical threshold.

[0064] Based on this, the main solution of the present application is: obtaining a first field and a second field to be analyzed for similarity, and obtaining a pre-constructed processing link weight graph, wherein the processing link weight graph includes field nodes and weight connection edges, the field nodes connected by the weight connection edges have an operation processing relationship, and the edge weight of the weight connection edge is set based on the operation processing relationship; identifying a processing path from the first field to the second field in the processing link weight graph; determining the field similarity between the first field and the second field according to the processing path, wherein the field similarity is positively correlated with the edge weight of the weight connection edge in the processing path.

[0065] After obtaining the first field and the second field to be analyzed for similarity, the present application identifies the processing path existing between the fields in the pre-constructed processing link weight graph, and quantifies the similarity according to the edge weight set based on the operation processing relationship in the path, effectively avoiding the misjudgment caused by surface differences such as non-standard naming, synonyms or abbreviations in traditional structural similarity methods, thereby significantly improving the accuracy and interpretability of the results. In terms of computational efficiency, there is no need for complex content calculation in high-dimensional vector space (such as cosine similarity), and the dependence on large NLP models or knowledge graphs with huge computational resource consumption is completely avoided. By transforming the problem into a path search and weight aggregation problem in the pre-constructed graph structure, the efficiency of the graph structure algorithm can be fully utilized, and the computational complexity is greatly reduced, so that the actual demand for large-scale, batch field comparison of massive data tables in the financial data warehouse can be quickly responded.

[0066] It should be noted that the execution subject of each embodiment of the field similarity analysis method of the present application can be a computing service device with data processing, network communication and program running functions, such as a server, a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, and each embodiment of the field similarity analysis method of the present application does not make specific limitation on this.

[0067] Based on this, the present application proposes a field similarity analysis method of the first embodiment, as shown in Figure 1 The field similarity analysis method includes the following steps S10-S30:

[0068] Step S10, obtaining a first field and a second field to be analyzed for similarity, and obtaining a pre-constructed processing link weight graph, wherein the processing link weight graph includes field nodes and weight connection edges, there is an operation processing relationship between the field nodes connected by the weight connection edges, and the edge weight of the weight connection edge is set based on the operation processing relationship;

[0069] The first field and the second field are fields whose similarity is to be analyzed. Further, the first field and the second field can be derived from different data tables.

[0070] The processing link weight graph can be a pre-constructed and stored metadata graph, which can be directly loaded from the storage area for use when needed. Specifically, the weight graph is a metadata network organized in a graph structure, where the nodes represent various fields in the data warehouse or business system, and the connection edges represent the operation processing relationship between the fields, such as conversion, connection, aggregation, etc. in SQL query. Each weight connection edge is assigned a quantified edge weight, which can be pre-set based on the strength, importance or frequency of the operation processing relationship it represents, for example, according to the complexity of the processing steps, the closeness of the data dependency or the strength of the business logic association, so as to lay a data foundation for subsequent path analysis and similarity calculation.

[0071] For example, in an application scenario, the pre-constructed processing link weight graph is as shown in Figure 2 Figure 2 ​The process of processing T1 table and T2 table to obtain T3 table, and processing T3 table and T4 table to obtain T5 table is shown, and <LEFT_JOIN>_1 table and <LEFT_JOIN>_2 table are virtual tables obtained in the operation process. Specifically, T1 table includes three field nodes C1, C2 and C3, T2 table includes three field nodes C4, C5 and C6, T3 table includes three field nodes R1, R2 and R3, T4 table includes three field nodes C7, C8 and C9, T5 table includes three field nodes R4, R5 and R6, <LEFT_JOIN>_1 table includes three field nodes V1, V2 and V3, and <LEFT_JOIN>_2 table includes three field nodes V4, V4 and V6. The values marked on each weight connection edge in the figure are the edge weights of the weight connection edges.

[0072] It should be noted that the first field is an upstream field, and the second field is a downstream field. The upstream field refers to a field that is a data source or an input item in a data processing link, and it is usually located at the beginning or a relatively early position of the processing flow. The downstream field refers to a derived field that is directly or indirectly dependent on the upstream field through one or more operation processing steps (such as conversion, connection, aggregation, etc.), and it is located at the end or a relatively late position of the processing flow.

[0073] In step S20, a processing path from the first field to the second field is identified in the processing link weight graph.

[0074] In the processing link weight graph, the processing path from the first field to the second field can be identified by using a graph search algorithm, such as a breadth-first search (BFS) or a depth-first search (DFS) algorithm. In the graph, the node corresponding to the first field is taken as the starting point, and the node corresponding to the second field is taken as the end point. The path is explored according to the directionality constraint of the edge, so that the processing path from the field node corresponding to the first field to the field node corresponding to the second field is found in the processing link weight graph.

[0075] For example, in an application scenario, the processing link weight graph is shown in FIG. 2. The first field is T1.C1, and the second field is T3.R1. The identified processing path T1.C1—>T3.R1 is shown in FIG. 3. Figure 3

[0076] In step S30, the field similarity between the first field and the second field is determined according to the processing path, and the field similarity is positively correlated with the edge weight of the weight connection edge in the processing path.

[0077] ​The final field similarity between the first field and the second field is determined according to the identified processing path. The similarity is a comprehensive quantitative value, which is positively correlated with the distribution of edge weights of the weighted connection edges in the path. The specific calculation method can be weighted average of all edge weights in the path, minimum value, or aggregation calculation through a monotonically increasing function. The core principle is that the higher the edge weight in the path, the more direct and close the operation processing relationship between the fields, and the higher the finally calculated field similarity. In this way, the objective processing blood relationship in the business logic is converted into a quantifiable similarity index, realizing efficient and accurate automatic analysis.

[0078] In the first embodiment of the field similarity analysis method, in the second embodiment of the field similarity analysis method, the same or similar contents as the above embodiment one can be referred to the above introduction, and will not be repeated hereinafter. On this basis, the processing path includes at least one unbranched processing path from the first field to the second field, and the step of determining the field similarity between the first field and the second field according to the processing path comprises:

[0079] Step A10, traversing each of the unbranched processing paths, multiplying the edge weights of each of the weighted connection edges in the unbranched processing path to obtain a single-path similarity, wherein the edge weights of each of the weighted connection edges are greater than or equal to zero or less than or equal to one;

[0080] According to the order of the weighted connection edges in the unbranched processing path, the edge weights of each weighted connection edge are multiplied in turn. For example, if a unbranched processing path contains weighted connection edges e1, e2 and e3, and their corresponding edge weights are w1, w2 and w3 respectively, then the single-path similarity Sp of the unbranched processing path is w1*w2*w3. And the value range of the edge weight is greater than or equal to zero or less than or equal to one, so as to ensure that the value of the single-path similarity is within a reasonable range, facilitating subsequent similarity calculation and comparison.

[0081] Step A20, calculating the average of all the single-path similarities to obtain the field similarity between the first field and the second field.

[0082] All the single-path similarity values are added and then divided by the total number of unbranched processing paths. Assuming that there are n unbranched processing paths, and their single-path similarities are Sp1, Sp2, … Spn respectively, then the field similarity S between the first field and the second field is (Sp1+Sp2+…+Spn) / n.

[0083] Exemplarily, in a specific application scenario, the processing path is as follows Figure 3If so, then the field similarity between T3.R1 and T1.C1 is (1*0.2+0.2*0.1) / 2=0.11.

[0084] In one possible implementation, the step of identifying a processing path from the first field to the second field in the processing link weight graph comprises:

[0085] Step B10, if the first field and the downstream field in the second field are multi-job processing fields, then each job processing window is traversed in turn.

[0086] Multi-job processing fields refer to fields that may be processed by multiple different jobs (e.g., different data processing tasks or processes) in the data processing process. Each job processing window refers to a partial or complete view of the processing link weight graph in a specific job, which is used to represent the operation and processing relationship between fields in the job. If the second field (downstream field) is a multi-job processing field, then each related job processing window needs to be traversed in turn to comprehensively identify all possible processing paths.

[0087] For example, in an application scenario, in the lineage analysis and impact assessment scenario of a typical financial data warehouse, the downstream field may be generated by multiple independent and periodic job tasks. For example, there may be a daily batch processing window, a real-time data stream processing window, and a dedicated processing window for specific business lines (such as credit and risk control). In different job processing windows, due to differences in business objectives, data processing frequency, and technology stack, the way to obtain the same target data table and the processing logic may be completely different. Therefore, corresponding to each independent job processing window, the processing link weight graph constructed and relied on is also logically different. Each window corresponds to a subgraph or graph version reflecting the specific job data processing logic. This embodiment can finally evaluate a more comprehensive and accurate global similarity across multiple processing modes between fields by traversing these windows in turn and performing path discovery and similarity calculation in the corresponding specific link graph.

[0088] It should be noted that if the first field and the downstream field in the second field are multi-job processing fields, but only the local similarity of the first field and the second field under a target job processing window is requested to be analyzed, or the first field and the downstream field in the second field are single-job processing fields, and the job processing window containing the first field and the second field is the target job processing window, the processing path from the first field to the second field can be identified based on the processing link weight graph under the target job window, and the field similarity between the first field and the second field is determined based on the processing path, that is, the local similarity of the first field and the second field under the target job processing window. On the contrary, if the first field and the downstream field in the second field are multi-job processing fields, and the global similarity of the first field and the second field under all job processing windows is requested to be analyzed, step B10 is performed.

[0089] Step B20, identifying a processing path from the first field to the second field based on the processing link weight graph under the job processing window;

[0090] For each job processing window, a graph search algorithm (such as breadth-first search or depth-first search) can be used to find a processing path from the first field to the second field in the processing link weight graph under the window. The processing path represents how the first field is converted into the second field through a series of operation processing relationships in the job.

[0091] The step of determining the field similarity between the first field and the second field according to the processing path comprises:

[0092] Step B30, determining the field similarity between the first field and the second field under each job processing window according to the processing path under each job processing window, so as to obtain the global similarity between the first field and the second field by calculating the average value of all the field similarities under all the job processing windows.

[0093] For the processing path under each job processing window, the field similarity under the job processing window can be calculated according to the method of steps A10 to A20. That is, for each branchless processing path, the edge weights of the weight connection edges in the path are multiplied to obtain a single-path similarity, and then the average value of all single-path similarities is calculated to obtain the field similarity under the job processing window.

[0094] After obtaining the field similarity for all job processing windows, these similarity values ​​are summed and divided by the total number of job processing windows to obtain the global similarity between the first and second fields. For example, if there are m job processing windows with field similarities S1, S2, ..., Sm, the global similarity Sglobal is (S1 + S2 + ... + Sm) / m. This method comprehensively considers the impact of different job processing paths on field similarity, resulting in a comprehensive and accurate global similarity assessment.

[0095] Based on the first and / or second embodiments of the field similarity analysis method, in the third embodiment of the field similarity analysis method, the content that is the same as or similar to the first and second embodiments described above can be referred to the above description and will not be repeated hereafter. Based on this, before the step of obtaining the pre-constructed processing link weight graph, the method further includes:

[0096] Step C10: Obtain a pre-constructed processing link logic graph, wherein the processing link logic graph includes field nodes and unweighted connection edges, and there are computational processing relationships between the field nodes connected by the unweighted connection edges;

[0097] The processing chain logic diagram is a pre-built graph structure used to represent the computational processing relationships between fields. This graph includes field nodes and unweighted connecting edges. Field nodes represent various fields in the financial data warehouse, and unweighted connecting edges represent the computational processing relationships between fields, but have not yet been assigned specific weight values. This logic diagram can be constructed through the analysis of financial business logic and data processing flows.

[0098] For example, in one application scenario, the partial graph structure of the pre-built processing link logic graph is as follows: Figure 5 As shown, Figure 5 This illustrates the process of processing tables T1 and T2 to obtain table T3.<LEFT_JOIN> Table _1 is a virtual table obtained during the operation. Specifically, table T1 includes three field nodes: C1, C2, and C3; table T2 includes three field nodes: C4, C5, and C6; and table T3 includes three field nodes: R1, R2, and R3.<LEFT_JOIN> Table _1 includes three field nodes: V1, V2, and V3. The calculation methods marked inside the unweighted connecting edges in the graph represent the calculation relationships between the corresponding fields of the two connected field nodes.

[0099] Step C20: Based on the computational processing relationship corresponding to each unweighted connection edge in the processing link logic diagram, set the edge weight of each unweighted connection edge;

[0100] A weight value can be assigned to each unweighted connection edge according to the complexity and importance of the operation processing relationship corresponding to the unweighted connection edge. For example, if a field is derived from another field by a complex calculation formula, the weight of the connection edge can be set lower; if a field is simply copied from another field, the weight of the connection edge can be set higher. The weight value can be a fixed value set empirically, or a dynamic value obtained through data analysis. The weight value can range from 0 to 1, where 0 represents no similarity and 1 represents complete similarity.

[0101] Step C30, after the edge weights of each unweighted connection edge in the processing link logical graph are set, a pre-constructed processing link weight graph is obtained.

[0102] The processing link weight graph is obtained by setting a specific edge weight for each unweighted connection edge based on the processing link logical graph. The weight graph completely represents the operation processing relationship between fields and its importance, and can be used for subsequent field similarity analysis. Once all edge weights are set, the weight graph can be stored for quick access and use when needed.

[0103] In one possible implementation, the step of setting the edge weight of each weighted connection edge in the processing link logical graph according to the operation processing relationship corresponding to the weighted connection edge includes:

[0104] Step D10, traversing each unweighted connection edge in the processing link logical graph, obtaining all operation processing relationships corresponding to the unweighted connection edge;

[0105] Starting from the starting node of the processing link logical graph, each unweighted connection edge is accessed in turn, and the field node pair corresponding to each connection edge is recorded. For each unweighted connection edge, all operation processing relationships corresponding to it are obtained. The operation processing relationship can be one or more operations, such as column operation operations such as case when, count, sum, or row operation operations such as where, group by. These operation processing relationships can be obtained by analyzing the data processing script or business logic, or extracted by a data lineage analysis tool, and the present embodiment does not make specific limitations thereto.

[0106] Step D20, obtaining the operation weight corresponding to each operation processing relationship according to a preset mapping relationship, wherein the preset mapping relationship is the corresponding relationship between different operation processing relationships and operation weights;

[0107] According to the preset mapping relationship, the operation weight corresponding to each operation processing relationship is obtained, specifically including: according to the pre-defined mapping relationship table, the operation weight corresponding to each operation processing relationship is searched. For example, the preset mapping relationship can be defined as: the weight of JOIN_ON operation is 0.98, the weight of sum operation is 0.1, the weight of substr operation is 0.2, and the weight of assignment operation is 1.

[0108] In step D30, all the operation weights are multiplied to obtain the edge weight of the edge without weight.

[0109] For each edge without weight, all the operation weights corresponding to the edge are multiplied. For example, if an edge without weight corresponds to two operation processing relationships, and the operation weights are 1 and 0.2 respectively, then the edge weight w of the edge without weight is 1*0.2. By multiplying the operation weights, the joint influence of multiple operation processing relationships on field similarity can be considered. The final edge weight value reflects the importance and complexity of the connection edge in the field processing link. The closer the edge weight value is to 1, the greater the contribution of the operation processing relationship of the connection edge to the field similarity.

[0110] For example, in an application scenario, a local processing link logic diagram is as shown in Figure 5 The weight of JOIN_ON operation is 0.98, the weight of sum operation is 0.1, the weight of substr operation is 0.2, the weight of concatenation operation is 0.2, and the weight of assignment operation is 1. After setting the weights of the part of the processing link logic diagram, a part of the processing link weight diagram as shown in Figure 6 is obtained. The weights corresponding to all edges without weight in the processing link logic diagram can be set in this way, so that a complete processing link weight diagram is constructed.

[0111] In a possible implementation, before the step of obtaining the pre-constructed processing link logic diagram, the method further includes:

[0112] In step E10, each SQL statement in the data processing job script is converted into a minimum processing unit SDP.

[0113] The data processing job script refers to a SQL script file used to execute a series of data processing tasks in a financial data warehouse, which contains multiple SQL statements, each statement realizing a specific data processing function, such as data extraction, conversion, loading, etc. SQL statement is a standard programming language for operating relational databases, which can query, insert, update, delete, etc.

[0114] Specifically, the data analysis system retrieves data processing job script files from the data warehouse's job scheduling system. Then, it uses an SQL parser to read each SQL statement in the script and performs syntax and semantic analysis on them. During parsing, key elements from the SQL statements are extracted, such as the field list in the SELECT clause, the table name in the FROM clause, and the conditional expressions in the WHERE clause.

[0115] Based on these key elements, the smallest processing unit, SDP, is constructed, and the processing logic of SQL statements is abstracted into the input-output relationships and data processing operations of the SDP.

[0116] Furthermore, to simplify the data processing model, the standard data processing model defines a unified minimum processing unit (SDP, Structured Data Processing). Each SDP is a standard data matrix operation, where one or two matrices are processed through standard row and column operations to output a new matrix.

[0117] For example, in one application scenario, a minimum processing unit (SDP) is as follows: Figure 4 As shown, the specific operation flow represented by this SDP is as follows: the C1 field of the source data matrix M1 affects the C1 field of the output data matrix M3 through certain column operations; the C2 field of the source data matrix M1 affects the C2 field of the output data matrix M3 through certain column operations; the C3 field of the source data matrix M1 and the C3 field of the source data matrix M2 affect the C3 field of the output data matrix M3 through certain column operations; and the C1 field of the source data matrix M1 affects the entire row of data in the output data matrix (M3) through certain row operations.

[0118] Step E20: Construct an SQL-level directed acyclic graph based on the syntactic operation relationships between the smallest processing units (SDPs);

[0119] Syntactic operation relations refer to the logical relationships between different data processing operations in SQL statements. SQL syntax, a concrete implementation based on relational algebra, describes the detailed data processing process. SDP's DAG simplifies the SQL representation, retaining the most important row and column operations while simplifying various complex expressions. Syntactic operation relations define operations such as projection, selection, union, difference, Cartesian product, and aggregation. An SQL-level Directed Acyclic Graph (DAG) is a data structure used to represent the dependencies and execution order between the smallest processing units (SDPs) in an SQL statement. In data warehouses, SQL statements are used to perform various complex data queries and processing operations.

[0120] Specifically, the syntax structure of the SQL statement is parsed to identify each data processing operation (such as the SELECT, JOIN, WHERE, etc. clauses of a table) and the syntax operation relationship between them. Based on these relationships, each data processing operation is converted into a minimum processing unit (SDP), and the connection order between them is determined. When building the SQL-level DAG graph, the SDP is used as a node in the graph, and a directed edge is added between the nodes according to the syntax operation relationship, indicating the flow of data from one SDP to another SDP.

[0121] For example, for a SQL statement containing two JOIN operations, the SELECT operations of the two tables are first converted into two SDP nodes, and then a directed edge is added according to the JOIN condition, indicating the flow of data from the two SDP nodes to the SDP node of the JOIN operation. In this process, the SQL parser is used to analyze the syntax tree of the SQL statement and extract each data processing operation and its relationship. For example, in a specific application scenario, the SQL-level directed acyclic graph constructed is as shown in Figure 5

[0122] In addition, by parsing the SQL statement to obtain basic data operations, the core of each data processing job in the data processing link is the SQL statement. By parsing the syntax structure of these SQL statements, the basic data operations therein are identified. For example, parsing the FROM clause in the SELECT statement can determine the data source table, parsing the WHERE clause can determine the data filtering condition, and parsing the GROUP BY clause can determine the data grouping method, etc. Using SQL syntax parsing technology, complex SQL statements are decomposed into a series of basic data operations.

[0123] Step E30, parsing the table dependency relationship between each of the SQL-level directed acyclic graphs, and constructing a job-level directed acyclic graph based on the table dependency relationship;

[0124] The table dependency relationship refers to the reference relationship between different data tables in the SQL-level directed acyclic graph, for example, the data of one table depends on the data of another table as input. The job-level directed acyclic graph (DAG) is an abstraction of the data processing flow from the perspective of the job, which represents the dependency relationship between multiple SQL-level DAG graphs, and a job usually contains an execution sequence of multiple SQL statements. In a data warehouse, a job is a basic task unit of data processing, used to complete a specific data processing goal.

[0125] ​For each SQL-level DAG graph, the data tables involved are parsed to determine the dependencies between these tables and tables in other SQL-level DAG graphs. For example, an output table of one SQL-level DAG graph can serve as an input table of another SQL-level DAG graph. Based on these table dependencies, multiple SQL-level DAG graphs are integrated together to build a job-level DAG graph. In the job-level DAG graph, a node represents a SQL-level DAG graph, and a directed edge indicates the data flow from the output of one SQL-level DAG graph to the input of another SQL-level DAG graph. By traversing the execution order of all SQL statements and table dependencies in the entire job, the relevant SQL-level DAG graphs are connected to form a complete job-level DAG graph.

[0126] As an optional implementation, by traversing all basic data operations and their corresponding SDPs in the entire data processing chain, the output data matrix of each SDP that depends on the output data matrix of a previous SDP can be determined. Based on these dependencies, corresponding nodes (SDPs) and directed edges are added to the DAG. For example, if the output data matrix of SDPA is one of the input data matrices of SDPB, a directed edge from SDPA to SDPB is added to the DAG. This process builds a preliminary graph structure reflecting the data processing order by analyzing the data flow.

[0127] In addition, it is necessary to check and eliminate loops to ensure the DAG property. A traversal check is performed on the preliminary constructed graph structure to find out whether there are loops. If a loop is found, the data processing chain needs to be re-examined and adjusted, and some data processing operations may need to be split or reorganized to eliminate the loop. For example, if SDPX depends on the output of SDPY, and SDPY depends on the output of SDPX, forming a loop, the data processing logic corresponding to these two SDPs needs to be re-evaluated. In this way, it is ensured that the finally constructed graph structure is a directed acyclic graph.

[0128] Step E40, constructing a link-level directed acyclic graph based on the job dependency relationship corresponding to the job-level directed acyclic graph;

[0129] Job dependency relationship refers to the execution order dependency relationship between different jobs in the job-level directed acyclic graph, for example, the execution of one job needs to wait for the output result of another job as input. The link-level directed acyclic graph (DAG) is an abstraction of the entire data processing chain from a more macro perspective, which represents the dependency relationship between multiple job-level DAG graphs and covers the complete data processing flow from the data source to the final data product. In a financial data warehouse, the link-level DAG graph is used to describe the dependency relationship between various jobs in the entire data warehouse, ensuring the correctness and efficiency of data processing.

[0130] Specifically, the preset tool analyzes the dependency relationship between the job-level DAG graphs, for example, the output table of job A is the input table of job B, so the execution of job B needs to wait for the completion of job A. According to these job dependencies, multiple job-level DAG graphs are integrated into a link-level DAG graph. In the link-level DAG graph, the node represents the job-level DAG graph, and the directed edge represents the execution order dependency between jobs. By traversing the execution order and dependency of all jobs in the entire data processing link, the related job-level DAG graphs are connected to form a complete link-level DAG graph.

[0131] Step E50, performing a data operation model operation on the link-level directed acyclic graph to generate an optimized link-level directed acyclic graph;

[0132] The data operation model operation is a general term for a series of optimization operations on the minimum processing unit (SDP) and its connection relationship in the directed acyclic graph (DAG). The data operation model operation includes pruning, moving up, splicing, etc. Pruning refers to cutting unnecessary data processing branches or redundant fields according to user requirements or data usage; moving up refers to advancing the position of certain SDP in the DAG to reduce data processing steps or improve the timeliness of data processing; splicing refers to merging or recombining multiple SDPs to generate more concise and efficient data processing logic.

[0133] Specifically, the link-level directed acyclic graph is first analyzed to identify the optimizable nodes and edges in the graph. For example, by checking the data dependency relationship and operation type between nodes, node pairs that can be fused are found, such as two consecutive processing units that perform simple operations on the same field, which can be merged into a composite operation unit. At the same time, by analyzing the data flow, redundant edges that do not change the essence of the data or have been covered by other paths are removed. Thus, an optimized target data processing link is generated.

[0134] Step E60, determining the optimized link-level directed acyclic graph as the pre-constructed processing link logical graph.

[0135] The optimized link-level directed acyclic graph is obtained, and the link-level directed acyclic graph is determined as the constructed processing link logical graph.

[0136] In addition, the present application also proposes a table similarity analysis method, which includes the following steps S100-S300 in the first embodiment of the table similarity analysis method:

[0137] Step S100, obtaining a first data table and a second data table to be analyzed for similarity;

[0138] Step S200, calculating the field similarity between any two fields in the first data table and the second data table, wherein the field similarity is determined by the field similarity analysis method as described in any of the above embodiments;

[0139] For each field in the first data table, the similarity is calculated with each field in the second data table. The field similarity between fields is determined by the field similarity analysis method as described above, for example, by processing the link weight graph. Specifically, a pre-constructed processing link weight graph is constructed or obtained, which includes field nodes and weight connection edges representing the operation processing relationship between fields and its weight. For the field F1i in the first data table and the field F2j in the second data table, the processing path from F1i to F2j is identified in the processing link weight graph. The multiplication value of the edge weight of all weight connection edges in each branchless processing path of the processing path is calculated to obtain the single path similarity. The average value of all single path similarities is calculated to obtain the field similarity between F1i and F2j. Repeat the above process to calculate the field similarity between all field pairs in the first data table and the second data table.

[0140] Step S300, determining the table similarity between the first data table and the second data table according to the field similarity, wherein the table similarity is positively correlated with the field similarity.

[0141] The table similarity is calculated according to all field similarities. For example, the average value of all field similarities can be calculated as a preliminary evaluation value of the table similarity. At the same time, the field similarity is weighted considering the importance of the field, such as assigning a higher weight to a key field (such as a primary key field or a business core field), and assigning a lower weight to a secondary field. A more accurate table similarity is obtained by weighted summation.

[0142] The final table similarity value reflects the similarity between the first data table and the second data table at the field level. The higher the table similarity, the more similar the two data tables are in structure and content.

[0143] Based on the first embodiment of the table similarity analysis method, in the second embodiment of the table similarity analysis method, the same or similar contents as the above embodiment one can be referred to the above introduction, and will not be repeated hereinafter. On this basis, the step of determining the table similarity between the first data table and the second data table according to the field similarity comprises:

[0144] Step A100, constructing an initial similarity matrix, wherein the number of rows of the initial similarity matrix is the total number of fields in the first data table, and the number of columns of the similarity matrix is the total number of fields in the second data table;

[0145] An initial similarity matrix is created, which has the same number of rows as the total number of fields in the first data table and the same number of columns as the total number of fields in the second data table. This matrix will be used to store the similarity values between fields in the two data tables.

[0146] Step A200, write each field similarity into the corresponding element position in the initial similarity matrix, and write a preset initial value into the remaining element positions, to obtain a target similarity matrix, wherein the similarity between the i-th field in the first data table and the j-th field in the second data table corresponds to the element position of the i-th row and the j-th column of the initial similarity matrix.

[0147] At this stage, according to the previously calculated field similarities, these values are filled into the corresponding positions of the initial similarity matrix. If there is no field relationship between some field pairs, i.e., there is no processing path between the field pairs, then a preset initial value is filled in these positions. This preset initial value can be a specific number representing the absence of a field relationship, for example, -1.

[0148] Step A300, traverse each row element value of the target similarity matrix, and calculate the average value of all target element values in each row element value, wherein the target element value is an element value that is not the preset initial value.

[0149] Traverse each row of the initial similarity matrix. For each row, only consider those elements that are not the preset initial value (i.e., field pairs for which the similarity has been calculated). Calculate the average value of these target element values, which reflects the average similarity between a field in the first data table and all related fields in the second data table.

[0150] Step A400, divide the sum of all the average values by the number of rows of the target similarity matrix to obtain the table similarity between the first data table and the second data.

[0151] Add all the average values obtained in step A300, and then divide by the number of rows of the target similarity matrix. This result is the table similarity between the first data table and the second data table, which integrates the similarity information of all field pairs in the two data tables and provides a comprehensive similarity evaluation for judging the similarity of the two data tables in structure and content.

[0152] For example, in an application scenario, the field link processing weight graph is as shown in Figure 2 The first data table is T5 and the second data table is T1, and the initial similarity matrix constructed is where s i-＞jwherein Sij represents the field or table similarity between i and j, and the initial value is set as -1. The target similarity matrix obtained by replacing the matrix with numerical values is The table similarity calculated by the target similarity matrix is

[0153] In a possible implementation, the step of calculating the field similarity between any two fields in the first data table and the second data table, the method further comprises:

[0154] Step B100, if the downstream data table in the first data table and the second data table is a multi-job processing table, then each job processing window is traversed in sequence;

[0155] Step B200, the field similarity between any two fields in the first data table and the second data table is calculated under the job processing window;

[0156] The step of determining the table similarity between the first data table and the second data table according to the field similarity comprises:

[0157] Step B300, the table similarity between the first data table and the second data table under each job processing window is determined according to the field similarity under each job processing window, so as to obtain the global similarity between the first data table and the second data table under all job processing windows.

[0158] Similar to the multi-job processing field, when the downstream data table in the first data table and the second data table is a multi-job processing table, and the global similarity between the first data and the second data under all job processing windows is requested to be analyzed, step B100 is executed to calculate the field similarity between any two fields in the first data table and the second data table under each job processing window based on the processing link weight graph under each job processing window, and then the table similarity under the job processing window is calculated.

[0159] After obtaining the table similarity under all job processing windows, the global similarity between the first data table and the second data table is obtained by adding the similarity values and dividing by the total number of job processing windows. For example, if there are m job processing windows, and the table similarity of each job processing window is S1, S2, … Sm, then the global similarity Sglobal is (S1+S2+…+Sm) / m. In this way, the influence of different job processing paths on the table similarity can be considered comprehensively, and a comprehensive and accurate global similarity evaluation result can be obtained.

[0160] Furthermore, an electronic device is provided in an embodiment of the present application. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the steps of the method described above.

[0161] Reference Figure 7 which shows a structural schematic diagram of an electronic device suitable for implementing an embodiment of the present application. The electronic device in the embodiment of the present application can also include, but is not limited to, a mobile terminal such as a mobile phone, a server, a notebook computer, a PDA (Personal Digital Assistant), a PAD (Portable Application Description), a PMP (Portable Media Player), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 7 The electronic device shown is only an example and should not bring any limitation to the function and use range of the embodiments of the present application.

[0162] As shown in Figure 7 , the electronic device can include a processing device 1001 (for example, a central processor, a graphics processor, and the like) which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, and the like; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; the storage device 1003 including, for example, a magnetic tape, a hard disk, and the like; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although an electronic device with various systems is shown in the figure, it should be understood that all the systems shown are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0163] In particular, according to the embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments of the present application are executed.

[0164] The electronic device provided by the embodiments of the present application adopts the field similarity analysis method in the above embodiments, and can solve the technical problem of how to realize efficient and high-accuracy field similarity analysis. Compared with the prior art, the electronic device provided by the present application has the same beneficial effects as the field similarity analysis method provided by the above embodiments, and other technical features in the electronic device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0165] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0166] The above describes only the specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0167] In addition, to achieve the above-mentioned purpose, the embodiments of the present application also provide a readable storage medium having computer readable program instructions (i.e. computer program) stored thereon, the computer readable program instructions being used to execute the field similarity analysis method in the above embodiments.

[0168] The computer readable storage medium provided by the embodiments of the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination of the above.

[0169] The above computer readable storage medium can be contained in an electronic device, or can exist separately without being assembled into an electronic device.

[0170] The above computer readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the steps of the above method.

[0171] Computer program code for carrying out operations of the present application can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as "C" or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0172] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0173] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0174] The computer readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., a computer program) for executing the field similarity analysis method described above, and can solve the technical problem of how to implement efficient and high-accuracy field similarity analysis. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the field similarity analysis method provided by the above-mentioned embodiments, which will not be repeated here.

[0175] In addition, the embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps of the method described above.

[0176] The computer program product specific embodiments of the present application are basically the same as the above-mentioned field similarity analysis method embodiments, which will not be repeated here.

[0177] It should be noted that in this paper, the term "include", "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or system. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0178] The above application embodiment serial numbers are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0179] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art can be embodied in the form of a software sensor. The computer software sensor is stored in a storage medium (such as ROM / RAM, magnetic disc, optical disc) as described above, and includes a plurality of instructions for making an electronic device (which can be a mobile phone, computer, server or network device, etc.) execute the method described in each embodiment of the present application.

[0180] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A field similarity analysis method, characterized in that, The field similarity analysis method includes the following steps: Obtain the first and second fields of similarity to be analyzed, and obtain a pre-constructed processing link weight graph, wherein the processing link weight graph includes field nodes and weight connection edges, there is a computational processing relationship between the field nodes connected by the weight connection edges, and the edge weight of the weight connection edges is set based on the computational processing relationship; Identify the processing path from the first field to the second field in the processing link weight graph; The field similarity between the first field and the second field is determined based on the processing path, wherein the field similarity is positively correlated with the edge weight of the weighted connecting edge in the processing path.

2. The field similarity analysis method as described in claim 1, characterized in that, The processing path includes at least one branchless processing path from the first field to the second field, and the step of determining the field similarity between the first field and the second field based on the processing path includes: Traverse each of the branchless processing paths and multiply the edge weights of the weighted connecting edges in the branchless processing path to obtain the single path similarity, wherein the edge weights of the weighted connecting edges are all greater than or equal to zero or less than or equal to one. Calculate the average of all the single-path similarities to obtain the field similarity between the first field and the second field.

3. The field similarity analysis method as described in claim 1, characterized in that, Before the step of obtaining the pre-constructed processing link weight map, the method further includes: Obtain a pre-constructed processing link logic graph, wherein the processing link logic graph includes field nodes and unweighted connection edges, and there are computational processing relationships between the field nodes connected by the unweighted connection edges; Based on the computational processing relationship corresponding to each unweighted connection edge in the processing link logic diagram, set the edge weight of each unweighted connection edge; After the edge weights of all unweighted connection edges in the processing link logic diagram are set, a pre-constructed processing link weight diagram is obtained.

4. The field similarity analysis method as described in claim 3, characterized in that, The step of setting the edge weight of each weighted connection edge according to the operation processing relationship corresponding to each weighted connection edge in the processing link logic diagram includes: Traverse each unweighted connection edge in the processing link logic graph to obtain all the operation and processing relationships corresponding to the unweighted connection edge; The computation weights corresponding to each computation processing relationship are obtained according to a preset mapping relationship, wherein the preset mapping relationship is the correspondence between different computation processing relationships and computation weights; Multiply all the calculated weights to obtain the edge weight of the unweighted connected edge.

5. The field similarity analysis method as described in claim 3, characterized in that, Before the step of obtaining the pre-constructed processing link logic diagram, the method further includes: Convert each SQL statement in the data processing job script into a minimum processing unit (SDP); Construct an SQL-level directed acyclic graph based on the syntactic operation relationships between the smallest processing units (SDPs); Analyze the table dependencies between each of the SQL-level directed acyclic graphs, and construct a job-level directed acyclic graph based on the table dependencies; Based on the job dependencies corresponding to the job-level directed acyclic graph, construct a link-level directed acyclic graph; Perform data computation model operations on the link-level directed acyclic graph to generate an optimized link-level directed acyclic graph; The optimized link-level directed acyclic graph is determined to be the pre-constructed processing link logic graph.

6. The field similarity analysis method as described in claim 1, characterized in that, The step of identifying the processing path from the first field to the second field in the processing link weight graph includes: If the downstream field in the first field and the second field is a multi-job processing field, then each job processing window is traversed sequentially; Based on the processing link weight graph under the job processing window, identify the processing path from the first field to the second field; The step of determining the field similarity between the first field and the second field based on the processing path includes: Based on the processing path under each of the job processing windows, the field similarity between the first field and the second field under each job processing window is determined. After obtaining the field similarity under all job processing windows, the average of all field similarities is calculated to obtain the global similarity between the first field and the second field.

7. A table similarity analysis method, characterized in that, The table similarity analysis method includes the following steps: Obtain the first and second data tables for the similarity analysis; Calculate the field similarity between any two fields in the first data table and the second data table, wherein the field similarity is determined using the field similarity analysis method as described in any one of claims 1 to 6; The table similarity between the first data table and the second data table is determined based on the similarity of each of the fields, wherein the table similarity is positively correlated with the similarity of each of the fields.

8. The field similarity analysis method as described in claim 7, characterized in that, The step of determining the table similarity between the first data table and the second data table based on the similarity of each of the fields includes: Construct an initial similarity matrix, wherein the number of rows in the initial similarity matrix is ​​equal to the total number of fields in the first data table, and the number of columns in the similarity matrix is ​​equal to the total number of fields in the second data table; The similarity of each field is written to the corresponding element position in the initial similarity matrix, and the preset initial value is written to the remaining element positions to obtain the target similarity matrix. The similarity between the i-th field in the first data table and the j-th field in the second data corresponds to the element position in the i-th row and j-th column of the initial similarity matrix. Traverse each row of the target similarity matrix and calculate the average of all target element values ​​in each row, wherein the target element values ​​are not the preset initial values. The table similarity between the first data table and the second data table is obtained by dividing the sum of all the average values ​​by the number of rows in the target similarity matrix.

9. The field similarity analysis method as described in claim 7, characterized in that, The method for calculating the field similarity between any two fields in the first data table and the second data table further includes: If the downstream data tables in the first data table and the second data table are multi-job processing tables, then each job processing window is traversed sequentially. Calculate the field similarity between any two fields in the first data table and the second data table under the job processing window; The step of determining the table similarity between the first data table and the second data table based on the similarity of each of the fields includes: Based on the similarity of each field under each job processing window, the table similarity between the first data table and the second data table under each job processing window is determined. After obtaining the table similarity under all job processing windows, the average value of all table similarities is calculated to obtain the global similarity between the first data table and the second data table.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 9.

Images