Data relationship graph construction method and device, equipment, and storage medium

Abstract

The application relates to a data relationship graph construction method, device and equipment and a storage medium, which comprises the following steps: obtaining target information of target structured data, classifying the target structured data according to themes according to the target information; for the target structured data of any theme, extracting numerical features and long text features for clustering analysis, generating a plurality of clustering families, obtaining relationship pairs of the target structured data in the plurality of clustering families, determining the data contact density of the relationship pairs according to a preset data correlation relationship level, and constructing a data relationship graph. The application classifies the relationship pairs by using the data correlation relationship level to construct the data relationship graph. This construction mode pays attention to the correlation relationship between the data itself content and important information in the forming process, and can be well applied to the correlation relationship full-amount analysis of massive data.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, specifically to a method, apparatus, device, and storage medium for constructing a data relationship graph. Background Technology

[0002] The relationships between structured data refer to the connections between different fields in two datasets and the business relationships between them. In relational databases, data relationships can be broadly categorized into one-to-many and many-to-many relationships, with further subdivisions including left join, right join, outer join, and inner join. In data association analysis, data relationships reflect the interdependence between something and other things; these relationships can be direct or indirect, such as the famous story of Walmart's "beer and diapers." To facilitate the analysis of various data relationships, data scientists have proposed algorithms such as Apriori, SETM, FP-tree, FP-G, FreeSpan, Prefixspan, and FP-growth to analyze the relationships between two datasets. In addition, to measure the correlation and closeness between two variables in a dataset, data scientists have also proposed methods such as Pearson correlation coefficient, Kendall correlation coefficient, and Spearman correlation coefficient. Currently, the mainstream techniques for data association analysis include correlation analysis, regression analysis, and cluster analysis. Based on the degree of association between data, the variables in two datasets can be categorized as having no relationship, having a vague relationship, having a strong but unclear relationship, or having a clear and measurable relationship.

[0003] However, most existing data relationship analysis methods only focus on the relationships between the content of the data itself, ignoring important information in the data warehouse formation process, resulting in the lack of a good open-source solution for full analysis of the relationships of massive amounts of data. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, device, and storage medium for constructing a data relationship graph, addressing the problem that existing data relationship analyses mostly focus only on the correlations between the content of the data itself, ignoring important information in the data warehouse formation process, resulting in a lack of good open-source solutions for full-scale analysis of the correlations of massive amounts of data. The specific technical solution is as follows:

[0005] According to a first aspect of the embodiments of this application, a method for constructing a data relationship graph is provided, the method comprising:

[0006] Obtain target information from the target structured data;

[0007] The target structured data is classified according to themes based on the target information;

[0008] For any given topic, the target structured data is used to extract numerical features and long text features for cluster analysis, generating multiple cluster families.

[0009] Obtain relation pairs about the target structured data from multiple clusters;

[0010] The data connection tightness of the relationship pairs is determined according to the preset data association hierarchy, and a data relationship graph is constructed.

[0011] Optionally, the target information includes the name information and business remarks information of the target structured data;

[0012] The step of classifying the target structured data according to the topic based on the target information includes:

[0013] The name information and business remarks information of the target structured data are segmented and part-of-speech identified;

[0014] The subject of the target structured data is determined based on the identification results and the target information;

[0015] The target structured data is classified according to its subject matter.

[0016] Optionally, before obtaining the relationship pairs regarding the target structured data in multiple clusters, the method further includes:

[0017] Calculate the similarity between any two target structured data for any given cluster family;

[0018] If the similarity between any two target structured data is greater than the similarity measure, then any two target structured data are determined to be a relation pair.

[0019] If the similarity between any two target structured data is less than the similarity measure, it is determined that the two target structured data are not a relation pair.

[0020] Optionally, the target structured data includes: target data packets, target datasets, and target data columns, wherein a target data packet includes multiple target datasets, and a target dataset includes multiple target data columns;

[0021] Before determining the data connection tightness of the relationship pairs according to the preset data association hierarchy and constructing the data relationship graph, the method further includes:

[0022] For any pair of relations, if the name similarity of the target dataset, the business scenario similarity of the target dataset, the name similarity of the target data packet, and the business scenario similarity of the target data packet are all greater than the similarity measure value, and the content of the target data columns of different target datasets has an inclusion relationship, then it is determined that any pair of relations has a first-level association relationship.

[0023] For any pair of relations, if the similarity of the target dataset name, the similarity of the target dataset business scenario, the similarity of the target data packet name, the similarity of the target data packet business scenario, the similarity of the target data column name topic, and the similarity of the target data column business note topic are all greater than the similarity measure value, then it is determined that any pair of relations has a secondary association relationship.

[0024] For any pair of relationships, if it is determined that the target structured data corresponding to the pair of relationships have a blood relationship and / or a derivative relationship, it is determined that the pair of relationships has a third-level association relationship;

[0025] For any pair of relationships, if it is determined that the target structured data corresponding to the pair of relationships has a business association, it is determined that the pair of relationships has a fourth-level association relationship.

[0026] For any pair of relationships, if it is determined that the target structured data corresponding to the pair of relationships has a trend of change, it is determined that the pair of relationships has a five-level association relationship.

[0027] The data association hierarchy is generated based on the first-level association, the second-level association, the third-level association, the fourth-level association, and the fifth-level association.

[0028] Optionally, determining the data connection tightness of the relationship pairs according to a preset data association hierarchy and constructing a data relationship graph includes:

[0029] Obtain the target similarity of the target structured data in the relationship pair. The target similarity includes: the name similarity of the target dataset, the business scenario similarity of the target dataset, the name similarity of the target data packet, the business scenario similarity of the target data packet, the name topic similarity of the target data column, and the business note topic similarity of the target data column.

[0030] The relationship level of the corresponding target data is determined based on the target similarity;

[0031] Obtain the calculation rules for the target data association hierarchy, and the first weight index for the target similarity;

[0032] The data closeness of the relationship pairs is generated based on the first weight index, the target similarity, and the calculation rule;

[0033] The data relationship graph is constructed based on the degree of data density.

[0034] Optionally, after obtaining the calculation rules for the target data association hierarchy and the first weight index for the target similarity, the method further includes:

[0035] If the target data association relationship is a first-level association relationship, then generate the intersection-union ratio of the relationship pairs;

[0036] Obtain the second weight index of the relation pair with respect to the intersection-union ratio;

[0037] Based on the second weight index, the intersection-union ratio of the relation pair, the first weight index, the target similarity, and the calculation rule, the data tightness of the relation pair in the first-level association relationship is generated.

[0038] According to a second aspect of the embodiments of this application, a data relationship graph construction apparatus is provided, the apparatus comprising:

[0039] The first acquisition module is used to acquire target information from the target structured data;

[0040] The first classification module is used to classify the target structured data according to the topic based on the target information;

[0041] The first generation module is used to extract numerical features and long text features from the target structured data for any topic, perform cluster analysis, and generate multiple cluster families.

[0042] The second acquisition module is used to acquire relationship pairs about the target structured data in multiple clusters;

[0043] The first construction module is used to determine the data connection tightness of the relationship pairs according to the preset data association hierarchy, and to construct a data relationship graph.

[0044] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising:

[0045] processor;

[0046] Memory used to store the processor's executable instructions;

[0047] The processor is configured to execute the instructions to implement the data relationship graph construction method as described in the first aspect.

[0048] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, which, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform the data relationship graph construction method as described in the first aspect of this application.

[0049] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0050] This invention acquires target information from target structured data; classifies the target structured data according to themes based on the target information; extracts numerical features and long text features for cluster analysis on target structured data of any theme, generating multiple clusters; obtains relationship pairs of target structured data within multiple clusters; determines the data connection tightness of relationship pairs according to a preset data association hierarchy, and constructs a data relationship graph. This invention initially determines relationship pairs of multiple target structured data using the name information, business notes, numerical features, and long text features of the target structured data, and then classifies the relationship pairs according to a set data association hierarchy. This constructs a data relationship graph. This construction method not only focuses on the relationships between the content of the data itself, but also on important information in the formation process of the target structured data, making it well-suited for full-scale analysis of the relationships in massive datasets.

[0051] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0053] Figure 1 This is a flowchart illustrating a data relationship graph construction method according to an exemplary embodiment;

[0054] Figure 2 This is a flowchart illustrating another method for constructing a data relationship graph according to an exemplary embodiment;

[0055] Figure 3 It is based on Figure 1 An exemplary embodiment illustrates a flowchart of step 105 in a data relationship graph construction method;

[0056] Figure 4 This is a block diagram illustrating a data relationship graph construction apparatus according to an exemplary embodiment;

[0057] Figure 5 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0058] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0059] The first embodiment of this application relates to a method for constructing a data relationship graph. Figure 1 This is a flowchart illustrating a data relationship graph construction method according to an exemplary embodiment, such as... Figure 1 As shown, it includes the following steps:

[0060] Step 101: Obtain the target information of the target structured data.

[0061] This invention constructs a relationship graph between target data packets, a relationship graph between target datasets, and a relationship graph between target data columns. These three types of relationship graphs are collectively referred to as data relationship graphs. Therefore, the target structured data includes: target data packets (target databases), target datasets (target data tables), and target data columns (target data variables). A target data packet includes multiple target datasets, a target dataset includes multiple target data columns, each target data packet includes business scenario notes for multiple data packets, each target dataset contains multiple target data columns, and each target data column (target data variable) consists of an array.

[0062] The data relationship graph in this embodiment of the invention is constructed based on the degree of business correlation and the tightness of data connection between target structured data. Therefore, the acquired target information includes the name information and business remarks information of the target structured data. Furthermore, since the target structured data includes target data packets (target databases), target datasets (target data tables), and target data columns (target data variables), the acquired target information includes the name information and business remarks information of the target data packets, the name information and business remarks information of the target datasets, and the name information and business remarks information of the target data columns (target data variables).

[0063] Step 102: Classify the target structured data according to the topic based on the target information.

[0064] In this embodiment of the invention, after obtaining the target information, the target information of the target structured data that is irrelevant to the business is first removed. Then, the name information and business remarks information of the target information are segmented and identified by part of speech, respectively. Based on the identification results, different parts of speech are classified. Then, based on the part-of-speech classification results and the target information (name information and business remarks information of the target structured data), the topics that can be generated are determined. The target structured data is then aggregated and classified according to the topics. The specific steps include:

[0065] The name information and business remarks information of the target structured data are segmented and part-of-speech identified;

[0066] The subject of the target structured data is determined based on the identification results and target information;

[0067] The target structured data is categorized according to its subject matter.

[0068] After part-of-speech tagging, different characters can be used to label the words. For example, let n be a common noun, s be a locative noun, t be a time, nh be a personal name, ns be a place name, nl be a locative word, nt be an organization name, nw be a work title, nz be other proper nouns, v be a common verb, vd be a verb-adverb, vn be a noun-verb, a be an adjective, a be a stative, ad be an adverb, d be an adverb, m be a quantifier, q be a classifier, r be a pronoun, i be an idiom, ni be an organization or group, j be an abbreviation, ws be a loanword, c be a conjunction, p be a preposition, wp be punctuation, u be an auxiliary word, etc. Stop words and words with parts of speech 'c', 'p', 'q', 'wp', and 'u' are discarded. Based on business analysis results, different parts of speech are classified as: ['n', 'nh', 'ns'], ['nl', 'v'], ['nt', 'r'], ['a', 'b'], ['ws', 'nz'], ['d', 'nd', 'm'], and ['i', 'ni', 'j']. Then, based on the part-of-speech classification results and target information, the generated topics about the target structured data are determined, and each topic is aggregated and classified.

[0069] Step 103: For any topic, extract numerical features and long text features from the target structured data and perform cluster analysis to generate multiple cluster families.

[0070] In this embodiment of the invention, different long text features are generated for different target structured data. Specifically, when constructing a relationship graph between target data packets, the long text features refer to the name information and business remarks information of the target dataset and the name information and business remarks information of the target data column in the target data packet. When constructing a relationship graph between target datasets, the long text features refer to the name information and business remarks information of the target data packet and the name information and business remarks information of the target data column. When constructing a relationship graph between target data columns, the long text features refer to the name information and business remarks information of the target data packet and the name information and business remarks information of the target dataset. It should be noted that when generating long text features, high-frequency and low-frequency data in the name information and business remarks information are also removed to make the results more reliable.

[0071] Furthermore, the numerical features extracted in this embodiment of the invention differ for different target structured data. Specifically, when constructing the relationship graph between target data packets, the numerical features refer to statistical features such as the number of target datasets contained in each target data packet, the number of target data columns contained in each target data packet, whether each target data packet has business remarks information, the number of target datasets containing business remarks information in each target data packet, and the number of target data columns containing business remarks information in each target data packet. When constructing the relationship graph between target datasets, the numerical features refer to statistical features such as the number of target data columns contained in each target dataset (data table), whether each target dataset (data table) has business remarks information, the number of target data columns containing business remarks information in each target dataset (data table), the number of target data columns containing business remarks information in each target dataset (data table), whether there is a unique identifier in the structure of the target dataset (data table), and the week (season) in which the target dataset (data table) was created. When constructing the relationship graph between target data columns, it is not necessary to extract numerical features.

[0072] In this embodiment of the invention, long text features are extracted using the doc2vec algorithm, and cluster analysis is performed on long text features and numerical features using kmeans, dbscan, and GSDMM respectively.

[0073] Step 104: Obtain the relationship pairs of the target structured data in multiple clusters.

[0074] In this embodiment of the invention, for the clustering results, the similarity of the aforementioned long text features is calculated pairwise for each cluster. Cosine similarity or edit distance can be used for similarity measurement; no specific limitation is made here. In this embodiment, a similarity threshold of 0.5 is set according to business characteristics. A similarity greater than 0.5 is considered a valid pairwise relationship for the target structured data. Based on this method, all relationship pairs related to the target structured data in multiple clusters are obtained.

[0075] Step 105: Determine the data connection tightness of the relationship pairs according to the preset data association hierarchy, and construct a data relationship graph.

[0076] In this embodiment of the invention, the data association hierarchy is designed based on the degree of business correlation and the tightness of data connection between the target structured data. The specific design process includes:

[0077] For any pair of relations, if the similarity of the names of the target datasets, the similarity of the business scenarios of the target datasets, the similarity of the names of the target data packets, and the similarity of the business scenarios of the target data packets are all greater than the similarity measure, and the content of the target data columns of different target datasets has an inclusion relationship, then any pair of relations is determined to have a first-level association relationship.

[0078] For any relation pair, if the similarity of the target dataset name, the similarity of the target dataset business scenario, the similarity of the target data packet name, the similarity of the target data packet business scenario, the similarity of the target data column name theme, and the similarity of the target data column business notes theme are all greater than the similarity measure value, then it is determined that any relation pair has a second-level association relationship.

[0079] For any pair of relations, if it is determined that the target structured data corresponding to the pair of relations have a blood relationship and / or a derivative relationship, it is determined that any pair of relations has a third-level association relationship;

[0080] For any pair of relationships, if it is determined that the target structured data corresponding to the pair of relationships has a business relationship, it is determined that any pair of relationships has a fourth-level relationship.

[0081] For any pair of relations, if it is determined that the target structured data corresponding to the pair of relations has a changing trend, it is determined that any pair of relations has a five-level association relationship;

[0082] Data relationship hierarchy is generated based on first-level, second-level, third-level, fourth-level, and fifth-level relationships.

[0083] In this embodiment of the invention, based on the target data packet (target database) relationship pairs, target dataset (target data table) relationship pairs, or target data column (target data variable) relationship pairs generated by clustering, a conclusion is drawn using the above-mentioned preset data association relationship hierarchy determination method, and a data relationship graph about the target structured data is constructed according to the conclusion.

[0084] This invention acquires target information from target structured data; classifies the target structured data according to themes based on the target information; extracts numerical features and long text features for cluster analysis on target structured data of any theme, generating multiple clusters; obtains relationship pairs of target structured data within multiple clusters; determines the data connection tightness of relationship pairs according to a preset data association hierarchy, and constructs a data relationship graph. This invention initially determines relationship pairs of multiple target structured data using the name information, business notes, numerical features, and long text features of the target structured data, and then classifies the relationship pairs according to a set data association hierarchy. This constructs a data relationship graph. This construction method not only focuses on the relationships between the content of the data itself, but also on important information in the formation process of the target structured data, making it well-suited for full-scale analysis of the relationships in massive datasets.

[0085] The second embodiment of this application relates to a method for constructing a data relationship graph. Figure 2 This is a flowchart illustrating another method for constructing a data relationship graph according to an exemplary embodiment, such as... Figure 2 As shown, it includes the following steps:

[0086] Step 201: Obtain target information from the target structured data.

[0087] Step 202: Classify the target structured data according to the topic based on the target information.

[0088] Step 203: For any topic, extract numerical features and long text features from the target structured data and perform cluster analysis to generate multiple cluster families.

[0089] It should be noted that, in the embodiments of the present invention, the above steps 201-203 are described in the preceding discussion and will not be repeated here.

[0090] Step 204: Calculate the similarity between any two target structured data for any cluster family.

[0091] In this embodiment of the invention, for any cluster, firstly, any two target structured data points are obtained, and then corresponding long text features are obtained, calculating the similarity between the two long text features. It should be noted that the target structured data points in the cluster need to be combined in any pairwise for calculation; that is, if the two target structured data points matched in the first instance are not a relational pair, other target structured data points will be matched separately.

[0092] Step 205: If the similarity between any two target structured data is greater than the similarity measure, determine that any two target structured data are a relation pair.

[0093] In this embodiment of the invention, cosine similarity or edit distance can be used for similarity measurement. Typically, the similarity value is set to 0.5. When the similarity between any two target structured data is greater than 0.5, they are determined to be a relation pair. Furthermore, after determining that any two target structured data are a relation pair, they are paired again with other target structured data in the same aggregate family, and their similarity is calculated.

[0094] Step 206: If the similarity between any two target structured data is less than the similarity measure, determine that any two target structured data are not a relation pair.

[0095] In this embodiment of the invention, the similarity measurement value is set to 0.5. When the similarity between any two target structured data is less than 0.5, it is determined that the two target structured data are not a relationship pair. At this time, the two target structured data can be paired with other target structured data in the aggregate family, and the similarity is calculated again.

[0096] Step 207: Obtain the relationship pairs of the target structured data in multiple clusters.

[0097] Step 208: Determine the data connection tightness of the relationship pairs according to the preset data association hierarchy, and construct a data relationship graph.

[0098] It should be noted that, in the embodiments of the present invention, steps 207-208 are described in the preceding discussion and will not be repeated here.

[0099] The third embodiment of this application relates to a method for constructing a data relationship graph. Figure 3 It is based on Figure 1 An exemplary embodiment shown illustrates a flowchart of step 105 in a data relationship graph construction method, as follows: Figure 3 As shown, it includes the following steps:

[0100] Step 301: Obtain the target similarity of the target structured data in the relationship pair.

[0101] In this embodiment of the invention, when determining the hierarchical relationship of the data association of the target structured data pairs, the target similarity of the target structured data is calculated. The target similarity includes: the name similarity of the target dataset, the business scenario similarity of the target dataset, the name similarity of the target data packet, the business scenario similarity of the target data packet, the name theme similarity of the target data column, and the business note theme similarity of the target data column.

[0102] Step 302: Determine the relationship level of the corresponding target data based on the target similarity.

[0103] In this embodiment of the invention, after obtaining the similarity, the first-level association relationship and the second-level association relationship can be verified. If neither is available, the blood relationship and derivative relationship of the relationship pair, as well as the business relationship relationship and the trend relationship relationship, will be obtained. The judgment will be made in order of priority from the first level to the fifth level, and finally the target data association relationship level of the target structured data will be determined.

[0104] Step 303: Obtain the calculation rules for the target data association hierarchy and the first weight index for target similarity.

[0105] In this embodiment of the invention, once the target data association level is determined, the corresponding calculation rules can be obtained. Since there are multiple similarities, the corresponding first weight index will also be obtained.

[0106] Step 304: Generate the data tightness of relation pairs based on the first weight index, target similarity, and calculation rules.

[0107] In this embodiment of the invention, the second-level association relationship can be directly generated by using the first weight index, target similarity, and calculation rules to determine the data tightness of the relationship pair. However, other levels of association relationships require additional parameters. For example, the first-level association relationship requires obtaining the intersection-union ratio (IUU) and its weight coefficient for the inclusion relationship. The specific steps include:

[0108] If the target data relationship is a first-level relationship, then generate the intersection-union ratio of the relationship pairs;

[0109] Obtain the second weight index of the intersection-union ratio in the relation pair;

[0110] The data density of relation pairs at the first-level association level is generated based on the second weight index, the intersection-union ratio of the relation pairs, the first weight index, the target similarity, and the calculation rules.

[0111] In addition, since third-level relationships have no similarity, the degree of kinship and / or derivative relationships of the relationship pairs are used to generate the data tightness; since fourth-level relationships also have no similarity, kinship and / or derivative relationships, the degree of business relevance of the relationship pairs is used to generate the data tightness; since fifth-level relationships also have no similarity, kinship and / or derivative relationships, and the business relevance is not obvious, the correlation of changing trends is used to generate the data tightness.

[0112] Example: The rules for calculating the data closeness of first-level relationships are as follows: Let the similarity between the target datasets (data tables) names be Stb1, the similarity between the target datasets (data tables) business scenarios be Stb2, the similarity between the target data packages (databases) business scenarios be Sdb1, and the similarity between the target data packages (databases) names be Sdb2. Let there be two target data columns (data variables): X1 = {x11, x12, ..., x1n}, where n is the number of instances contained in target data column X1, and X2 = {x21, x22, ..., x2m}, where m is the number of instances contained in target data column X2. The intersection-union ratio of X1 and X2 is... That is, the intersection of X1 and X2 divided by the union of X1 and X2. Based on the above definition, the present invention defines the data tightness measurement formula (1) for first-level association as follows:

[0113] (1)

[0114] Where γ* is the weight index for adjusting each condition, with a value range of [0,1], Px has a value range of [0,1], and S* has a value range of [0,1].

[0115] The calculation rules for the data tightness of secondary associations are as follows: Let the similarity of the target data packets (databases) business scenarios between the relationship pairs to be measured be Sdb1, the similarity of the dataset (data table) names be Sdb2, the similarity of the dataset (data table) business scenarios be Stb1, and the similarity of the dataset (data table) names be Stb2; the similarity of the data column (data column) names be Scol1, and the similarity of the data column (data column) business remarks be Scol2. Based on the above definition, the present invention defines the data tightness measurement formula (2) for secondary associations as follows:

[0116] (2)

[0117] Where γ* is the weight index for adjusting each condition, with a value range of [0,1], and S* has a value range of [0,1].

[0118] The rules for calculating the closeness of data in a three-level relationship are as follows: Let the closeness of the blood relationship between two relationship pairs be Pblan, where Pblan is 0 if there is a blood relationship and 1 if there is no blood relationship. The closeness of the derived relationship is Pderive, where Pderive is 1 if there is a derived relationship and 0 if there is no derived relationship.

[0119] The calculation rule for the data closeness of the fourth-level association is as follows: Let Pbus be the closeness of business interactions or business associations between two datasets. A business association is 1, and no business association is 0.

[0120] The degree of closeness of a five-level association relationship: Let the distributions of the datasets for two relationship pairs be X1 and X2. This invention selects commonly used methods to measure the relationship between the distributions of two data points, such as Pearson correlation coefficient (suitable for quantitative data and data that follows a normal distribution), Spearman correlation coefficient (used when data does not follow a normal distribution), Kendall's tau-b correlation coefficient (for ordinal categorical variables), KL divergence, univariate regression, and multivariate regression.

[0121] Step 305: Construct a data relationship graph based on the degree of data density.

[0122] This invention constructs a data relationship graph by acquiring the density of data, wherein there are corresponding data association levels between target data packets, between target datasets, and between target data columns. A data relationship graph is constructed as a whole from the target structured data of different objects.

[0123] The fourth embodiment of this application relates to a data relationship graph construction apparatus. Figure 4 This is a flowchart illustrating a data relationship graph construction apparatus according to an exemplary embodiment, such as... Figure 4 As shown, it includes the following steps:

[0124] The first acquisition module 401 is used to acquire target information of the target structured data.

[0125] The first classification module 402 is used to classify the target structured data according to the topic based on the target information.

[0126] The first generation module 403 is used to extract numerical features and long text features from target structured data for any topic, perform cluster analysis, and generate multiple cluster families.

[0127] The second acquisition module 404 is used to acquire relationship pairs of the target structured data in multiple clusters.

[0128] The first construction module 405 is used to determine the data connection tightness of the relationship pairs according to the preset data association hierarchy, and to construct a data relationship graph.

[0129] Optionally, the target information includes the name of the target structured data and business remarks.

[0130] The first classification module 402 also includes:

[0131] The identification submodule is used to perform word segmentation and part-of-speech recognition on the name information and business notes of the target structured data.

[0132] The first determination submodule is used to determine the subject of the target structured data based on the recognition results and target information.

[0133] The classification submodule is used to classify the target structured data according to the subject matter of the target structured data.

[0134] Optionally, the data relationship graph construction device also includes:

[0135] The calculation module is used to calculate the similarity between any two target structured data for any cluster family.

[0136] The first determination module is used to determine that any two target structured data are a relation pair if the similarity between them is greater than the similarity measure.

[0137] The second determination module is used to determine that any two target structured data are not a relation pair if the similarity between any two target structured data is less than the similarity measure.

[0138] Optionally, the target structured data includes: target data packets, target datasets, and target data columns, wherein a target data packet includes multiple target datasets, and a target dataset includes multiple target data columns.

[0139] The data relationship graph construction device also includes:

[0140] The third determination module is used to determine that any relation pair has a first-level association relationship if the similarity of the target dataset name, the similarity of the target dataset business scenario, the similarity of the target data packet name, and the similarity of the target data packet business scenario are all greater than the similarity measure value, and the target data column content of different target datasets has an inclusion relationship.

[0141] The fourth determination module is used to determine whether any relation pair has a secondary association relationship if the similarity of the target dataset name, the target dataset business scenario, the target data packet name, the target data packet business scenario, the target data column name topic, and the target data column business note topic are all greater than the similarity measure value.

[0142] The fifth determination module is used to determine whether any relationship pair has a third-level association relationship, provided that the target structured data corresponding to the relationship pair has a blood relationship and / or a derivative relationship.

[0143] The sixth determination module is used to determine whether any relationship pair has a four-level association relationship, provided that the target structured data corresponding to the relationship pair has a business relationship.

[0144] The seventh determination module is used to determine, for any pair of relationships, that there is a correlation between the target structured data corresponding to the pair of relationships and the existence of a changing trend, whether or not there is a five-level association relationship between the pair of relationships.

[0145] The second generation module is used to generate data relationship hierarchy based on first-level, second-level, third-level, fourth-level and fifth-level relationships.

[0146] Optionally, the first building module also includes:

[0147] The first acquisition submodule is used to acquire the target similarity of the target structured data in the relation pair. The target similarity includes: the name similarity of the target dataset, the business scenario similarity of the target dataset, the name similarity of the target data packet, the business scenario similarity of the target data packet, the name topic similarity of the target data column, and the business note topic similarity of the target data column.

[0148] The second determination submodule is used to determine the relationship hierarchy of the corresponding target data based on the target similarity.

[0149] The second acquisition submodule is used to acquire the calculation rules for the target data association hierarchy, as well as the first weight index for target similarity.

[0150] The first generation submodule is used to generate the data tightness of relation pairs based on the first weight index, target similarity, and calculation rules.

[0151] The first construction submodule is used to build a data relationship graph based on the density of the data.

[0152] The second generation submodule is used to generate the intersection-union ratio of relation pairs if the target data relationship level is a first-level relationship.

[0153] The third acquisition submodule is used to obtain the second weight index of the intersection-union ratio in the relation pair.

[0154] The third generation submodule is used to generate the data tightness of relation pairs in the first-level association relationship based on the second weight index, the intersection-union ratio of the relation pairs, the first weight index, the target similarity, and the calculation rules.

[0155] This invention acquires target information from target structured data; classifies the target structured data according to themes based on the target information; extracts numerical features and long text features for cluster analysis on target structured data of any theme, generating multiple clusters; obtains relationship pairs of target structured data within multiple clusters; determines the data connection tightness of relationship pairs according to a preset data association hierarchy, and constructs a data relationship graph. This invention initially determines relationship pairs of multiple target structured data using the name information, business notes, numerical features, and long text features of the target structured data, and then classifies the relationship pairs according to a set data association hierarchy. This constructs a data relationship graph. This construction method not only focuses on the relationships between the content of the data itself, but also on important information in the formation process of the target structured data, making it well-suited for full-scale analysis of the relationships in massive datasets.

[0156] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0157] The fifth embodiment of this application relates to an electronic device, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement any data relationship graph construction method.

[0158] Figure 5 This is a block diagram illustrating an electronic device 1400 according to an exemplary embodiment. For example, the electronic device 1400 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0159] Reference Figure 5 The electronic device 1400 may include one or more of the following components: processing component 1402, memory 1404, power supply component 1406, multimedia component 1408, audio component 1410, input / output interface 1412, sensor component 1414, and communication component 1416.

[0160] Processing component 1402 typically controls the overall operation of device 1400, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 1402 may include one or more processors 1420 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1402 may include one or more modules to facilitate interaction between processing component 1402 and other components. For example, processing component 1402 may include a multimedia module to facilitate interaction between multimedia component 1408 and processing component 1402.

[0161] Memory 1404 is configured to store various types of data to support the operation of device 1400. Examples of this data include instructions for any application or method operating on device 1400, contact data, phonebook data, messages, pictures, videos, etc. Memory 1404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0162] Power supply component 1406 provides power to various components of electronic device 1400. Power supply component 1406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1400.

[0163] Multimedia component 1408 includes a screen that provides an output interface between the electronic device 1400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1408 includes a front-facing camera and / or a rear-facing camera. When the electronic device 1400 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0164] Audio component 1410 is configured to output and / or input audio signals. For example, audio component 1410 includes a microphone (MIC) configured to receive external audio signals when electronic device 1400 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1404 or transmitted via communication component 1416. In some embodiments, audio component 1410 also includes a speaker for outputting audio signals.

[0165] Input / output interface 1412 provides an interface between processing component 1402 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0166] Sensor assembly 1414 includes one or more sensors for providing state assessments of various aspects of electronic device 1400. For example, sensor assembly 1414 may detect the on / off state of electronic device 1400, the relative positioning of components such as the display and keypad of electronic device 1400, changes in position of electronic device 1400 or a component of electronic device 1400, the presence or absence of user contact with electronic device 1400, orientation or acceleration / deceleration of electronic device 1400, and temperature changes of electronic device 1400. Sensor assembly 1414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1414 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1414 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0167] Communication component 1416 is configured to facilitate wired or wireless communication between electronic device 1400 and other devices. Electronic device 1400 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 1416 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1416 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0168] In an exemplary embodiment, the electronic device 1400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0169] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1404 including instructions, which can be executed by a processor 1420 of an electronic device 1400 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0170] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0171] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that alternative embodiments can be devised by those skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The invention is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for constructing a data relationship graph, characterized in that, The method includes: Obtain target information from the target structured data; the target information includes the name information and business remarks information of the target structured data; The target structured data is classified according to themes based on the target information; For any given topic, the target structured data is subjected to cluster analysis by extracting numerical features and long text features to generate multiple cluster families; the long text features include the name information and business remarks information of the target structured data. Obtain relation pairs about the target structured data from multiple clusters; The data connection tightness of the relationship pairs is determined according to a preset data association hierarchy, and a data relationship graph is constructed, including: Obtain the target similarity of the target structured data in the relationship pair. The target similarity includes: the name similarity of the target dataset, the business scenario similarity of the target dataset, the name similarity of the target data packet, the business scenario similarity of the target data packet, the name topic similarity of the target data column, and the business note topic similarity of the target data column. The relationship level of the corresponding target data is determined based on the target similarity; Obtain the calculation rules for the target data association hierarchy, and the first weight index for the target similarity; The data closeness of the relationship pairs is generated based on the first weight index, the target similarity, and the calculation rule; The data relationship graph is constructed based on the degree of data density.

2. The method according to claim 1, characterized in that, The step of classifying the target structured data according to the topic based on the target information includes: The name information and business remarks information of the target structured data are segmented and part-of-speech identified; The subject of the target structured data is determined based on the identification results and the target information; The target structured data is classified according to its subject matter.

3. The method according to claim 1, characterized in that, Before obtaining the relationship pairs of the target structured data in multiple clusters, the method further includes: Calculate the similarity between any two target structured data for any given cluster family; If the similarity between any two target structured data is greater than the similarity measure, then any two target structured data are determined to be a relation pair. If the similarity between any two target structured data is less than the similarity measure, it is determined that the two target structured data are not a relation pair.

4. The method according to claim 1, characterized in that, The target structured data includes: target data packets, target datasets, and target data columns, wherein one target data packet includes multiple target datasets, and one target dataset includes multiple target data columns; Before determining the data connection tightness of the relationship pairs according to the preset data association hierarchy and constructing the data relationship graph, the method further includes: For any pair of relations, if the name similarity of the target dataset, the business scenario similarity of the target dataset, the name similarity of the target data packet, and the business scenario similarity of the target data packet are all greater than the similarity measure value, and the content of the target data columns of different target datasets has an inclusion relationship, then it is determined that any pair of relations has a first-level association relationship. For any pair of relations, if the similarity of the target dataset name, the similarity of the target dataset business scenario, the similarity of the target data packet name, the similarity of the target data packet business scenario, the similarity of the target data column name topic, and the similarity of the target data column business note topic are all greater than the similarity measure value, then it is determined that any pair of relations has a secondary association relationship. For any pair of relationships, if it is determined that the target structured data corresponding to the pair of relationships have a blood relationship and / or a derivative relationship, it is determined that the pair of relationships has a third-level association relationship; For any pair of relationships, if it is determined that the target structured data corresponding to the pair of relationships has a business association, it is determined that the pair of relationships has a fourth-level association relationship. For any pair of relationships, if it is determined that the target structured data corresponding to the pair of relationships has a trend of change, it is determined that the pair of relationships has a five-level association relationship. The data association hierarchy is generated based on the first-level association, the second-level association, the third-level association, the fourth-level association, and the fifth-level association.

5. The method according to claim 1, characterized in that, After obtaining the calculation rules for the target data association hierarchy and the first weight index for the target similarity, the method further includes: If the target data association relationship is a first-level association relationship, then generate the intersection-union ratio of the relationship pairs; Obtain the second weight index of the relation pair with respect to the intersection-union ratio; Based on the second weight index, the intersection-union ratio of the relation pair, the first weight index, the target similarity, and the calculation rule, the data tightness of the relation pair in the first-level association relationship is generated.

6. A data relationship graph construction device, characterized in that, include: The first acquisition module is used to acquire target information from the target structured data; The target information includes the name information and business remarks information of the target structured data; The first classification module is used to classify the target structured data according to the topic based on the target information; The first generation module is used to extract numerical features and long text features from the target structured data for any topic, perform cluster analysis, and generate multiple cluster families; the long text features include the name information and business remarks information of the target structured data; The second acquisition module is used to acquire relationship pairs about the target structured data in multiple clusters; The first construction module is used to determine the data connection tightness of the relationship pairs according to the preset data association relationship hierarchy, and to construct a data relationship graph; The first building module also includes: The first acquisition submodule is used to acquire the target similarity of the target structured data in the relation pair. The target similarity includes: the name similarity of the target dataset, the business scenario similarity of the target dataset, the name similarity of the target data packet, the business scenario similarity of the target data packet, the name topic similarity of the target data column, and the business note topic similarity of the target data column. The second determining submodule is used to determine the target data association relationship level corresponding to the relationship pair based on the target similarity; The second acquisition submodule is used to acquire the calculation rules of the target data association hierarchy and the first weight index of the target similarity; The first generation submodule is used to generate the data closeness of the relationship pair based on the first weight index, the target similarity and the calculation rule; The first construction submodule is used to construct the data relationship graph based on the density of the data.

7. The apparatus according to claim 6, characterized in that, The first classification module also includes: The identification submodule is used to perform word segmentation and part-of-speech recognition on the name information and business remarks information of the target structured data; The first determining submodule is used to determine the topic of the target structured data based on the identification results and the target information; The classification submodule is used to classify the target structured data according to the subject of the target structured data.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to execute the instructions to implement the data relationship graph construction method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the mobile terminal, the mobile terminal is able to perform the data relationship graph construction method as described in any one of claims 1 to 5.

Images