Method, device, equipment, medium and product for determining blood relationship of data table

Abstract

The present application belongs to the field of financial technology or other related fields, and specifically provides a method, apparatus, device, medium and product for determining blood relationship of a data table, the method comprising: obtaining a first intermediate parameter corresponding to the first data table whose blood relationship is to be determined and a second intermediate parameter corresponding to the second data table; inputting the first data table name, the first field name, the second data table name and the second field name into a word vector model trained to convergence, generating a corresponding first table vector and a first field vector, as well as a corresponding second table vector and a second field vector; inputting the first table vector and the second table vector into a multi-layer perceptron model, generating a blood relationship determination result corresponding to the table name; if a blood relationship exists, generating a blood relationship determination result corresponding to the field name according to the first field vector, the second field vector and a preset similarity determination algorithm. The method of the present application improves the efficiency of determining blood relationships of data tables.

Description

Technical Field

[0001] The present application relates to the field of financial technology or other related fields, and in particular to a method, device, equipment, medium and product for determining blood relationship of a data table. Background Art

[0002] With the advent of the big data era, data is constantly increasing in size, complexity, and diversity, leading to numerous challenges in data management and analysis. For example, data sources are unclear, data changes cannot be traced, and data impacts are unpredictable. These issues can lead to decreased data quality and threats to data security. To address these issues, it is necessary to mine data tables for lineage relationships. This involves identifying the source, flow, and destination of data within a table, as well as the dependencies and transformations between data.

[0003] Currently, manual analysis is commonly used to determine the relationship between data tables. This method relies on business knowledge and experience to determine the relationship between data tables by checking the database. This method requires a lot of time and effort, and therefore has a low efficiency in determining the relationship between data tables. Summary of the Invention

[0004] The present application provides a method, device, equipment, medium and product for determining blood relationship in a data table, which is used to solve the problem that the current manual analysis method requires a lot of time and energy, resulting in low efficiency in determining blood relationship in a data table.

[0005] In a first aspect, the present application provides a method for determining blood relationships in a data table, comprising:

[0006] Obtaining a first intermediate parameter corresponding to a first data table for which blood relationship determination is to be made and a second intermediate parameter corresponding to a second data table; the first intermediate parameter includes a name of the first data table and a name of a first field included in the first data table; the second intermediate parameter includes a name of the second data table and a name of a second field included in the second data table;

[0007] Input the first data table name, the first field name, the second data table name, and the second field name into the word vector model trained to convergence to generate the corresponding first table vector and the first field vector, as well as the corresponding second table vector and the second field vector; the first table vector is the vector corresponding to the first data table name, the first field vector is the vector corresponding to the first field name, the second table vector is the vector corresponding to the second data table name, and the second field vector is the vector corresponding to the second field name;

[0008] Inputting the first table vector and the second table vector into a multi-layer perceptron model trained to convergence, generating a result of determining the blood relationship between the first data table and the second data table corresponding to the table name;

[0009] If the result of determining the blood relationship corresponding to the table names between the first data table and the second data table is that there is a blood relationship, then a result of determining the blood relationship corresponding to the field names between the first data table and the second data table is generated based on the first field vector, the second field vector and the preset similarity determination algorithm.

[0010] Furthermore, in the above method, obtaining the first intermediate parameter corresponding to the first data table for determining the blood relationship and the second intermediate parameter corresponding to the second data table includes:

[0011] Obtaining a first data table and a second data table of the blood relationship to be determined;

[0012] Extracting data table name data and data table field data corresponding to the first data table and the second data table;

[0013] Convert the data table name data and data table field data corresponding to the first data table into text format to generate first data table name text and first data table field name text;

[0014] Convert the data table name data and data table field data corresponding to the second data table into text format to generate a second data table name text and a second data table field name text;

[0015] The first data table name text, the first data table field name text, the second data table name text and the second data table field name text are preprocessed respectively to generate the first data table name, the first field name, the second data table name and the second field name.

[0016] Furthermore, in the above method, the preprocessing of the first data table name text, the first data table field name text, the second data table name text, and the second data table field name text respectively to generate the first data table name, the first field name, the second data table name, and the second field name includes:

[0017] Performing stop word removal preprocessing and stem extraction preprocessing on the first data table name text and the first data table field name text to generate the first data table name and the first field name;

[0018] The second data table name text and the second data table field name text are pre-processed by removing stop words and extracting stems to generate the second data table name and the second field name.

[0019] Furthermore, in the method described above, the word vector model includes a word2vec model, a preset table vector algorithm, and a preset field vector algorithm;

[0020] The step of inputting the first data table name, the first field name, the second data table name, and the second field name into a word vector model trained to convergence to generate a corresponding first table vector and a first field vector, as well as a corresponding second table vector and a second field vector includes:

[0021] Split the first data table name and the first field name into multiple words using a word2vec model, and determine the total number of words corresponding to all the words;

[0022] Use word2vec model to convert the words into corresponding word vectors;

[0023] Inputting all the word vectors and the total number of words into a preset table vector algorithm to generate a corresponding first table vector;

[0024] Inputting the word vector corresponding to the first field name into a preset field vector algorithm to generate a corresponding first field vector;

[0025] The second data table name and the second field name are vectorized based on the word vector model to generate corresponding second table vectors and second field vectors.

[0026] Furthermore, in the above method, the step of inputting the first table vector and the second table vector into a multi-layer perceptron model trained to convergence to generate a result of determining the blood relationship between the first data table and the second data table corresponding to the table names includes:

[0027] The multilayer perceptron model is used to generate a probability corresponding to a table name blood relationship category of the first data table relative to the second data table based on the first table vector and the second table vector; the table name blood relationship category includes upstream blood relationship, downstream blood relationship, and no blood relationship;

[0028] If the probability corresponding to the upstream blood relationship is greater than the probability corresponding to the downstream blood relationship, and the probability corresponding to the upstream blood relationship is greater than the probability corresponding to no blood relationship, then the blood relationship corresponding to the table names between the first data table and the second data table is determined to be an upstream blood relationship between the first data table and the second data table;

[0029] If the probability corresponding to the downstream blood relationship is greater than the probability corresponding to the upstream blood relationship, and the probability corresponding to the downstream blood relationship is greater than the probability corresponding to no blood relationship, then the blood relationship corresponding to the table names between the first data table and the second data table is determined to be a downstream blood relationship between the first data table and the second data table;

[0030] If the probability corresponding to no blood relationship is greater than the probability corresponding to the upstream blood relationship, and the probability corresponding to no blood relationship is greater than the probability corresponding to the downstream blood relationship, then the blood relationship corresponding to the table name between the first data table and the second data table is determined to be that the first data table has no blood relationship with the second data table.

[0031] Furthermore, in the above method, the preset similarity determination algorithm is a vector cosine similarity algorithm;

[0032] The generating, based on the first field vector, the second field vector, and a preset similarity determination algorithm, a result of determining the blood relationship corresponding to the field names between the first data table and the second data table includes:

[0033] Inputting the first field vector and the second field vector into the vector cosine similarity algorithm to generate corresponding cosine similarities;

[0034] If the cosine similarity is greater than a preset threshold, determining that the blood relationship corresponding to the field names between the first data table and the second data table exists is a blood relationship;

[0035] If the cosine similarity is less than or equal to a preset threshold, it is determined that the blood relationship corresponding to the field names between the first data table and the second data table does not exist.

[0036] Furthermore, the method as described above, before inputting the first table vector and the second table vector into a multi-layer perceptron model trained to convergence to generate a result of determining the blood relationship corresponding to the table names between the first data table and the second data table, further includes:

[0037] Obtaining a training sample, wherein the training sample includes: the actual blood relationship between each test data table, the data table name and data table field name corresponding to the test data table;

[0038] Inputting the training samples into a preset multi-layer perceptron model to train the preset multi-layer perceptron model;

[0039] Determining whether the preset multilayer perceptron model satisfies a preset convergence condition based on the test kinship relationship between the test data tables output by the preset multilayer perceptron model and the actual kinship relationship;

[0040] If the preset multilayer perceptron model satisfies the convergence condition, the preset multilayer perceptron model that satisfies the convergence condition is determined as the preset multilayer perceptron model trained to convergence.

[0041] A second aspect of the present application provides a device for determining blood relationship in a data table, comprising:

[0042] An acquisition module, configured to acquire a first intermediate parameter corresponding to a first data table for which a blood relationship is to be determined and a second intermediate parameter corresponding to a second data table; the first intermediate parameter includes a name of a first data table and a name of a first field included in the first data table; the second intermediate parameter includes a name of a second data table and a name of a second field included in the second data table;

[0043] A first generation module is configured to input the first data table name, the first field name, the second data table name, and the second field name into a word vector model trained to convergence, and generate a corresponding first table vector and first field vector, as well as a corresponding second table vector and second field vector; the first table vector is a vector corresponding to the first data table name, the first field vector is a vector corresponding to the first field name, the second table vector is a vector corresponding to the second data table name, and the second field vector is a vector corresponding to the second field name;

[0044] A second generating module is configured to input the first table vector and the second table vector into a multi-layer perceptron model trained to convergence, and generate a result of determining the blood relationship between the first data table and the second data table corresponding to the table names;

[0045] The third generation module is used to generate a blood relationship determination result corresponding to the field name between the first data table and the second data table based on the first field vector, the second field vector and a preset similarity determination algorithm if the blood relationship determination result corresponding to the table name between the first data table and the second data table is that there is a blood relationship.

[0046] Furthermore, in the above-mentioned device, the acquisition module is specifically configured to:

[0047] Obtain the first data table and the second data table for which the blood relationship is to be determined; extract the data table name data and data table field data corresponding to the first data table and the second data table; convert the data table name data and data table field data corresponding to the first data table into text format to generate the first data table name text and the first data table field name text; convert the data table name data and data table field data corresponding to the second data table into text format to generate the second data table name text and the second data table field name text; pre-process the first data table name text, the first data table field name text, the second data table name text and the second data table field name text respectively to generate the first data table name, the first field name, the second data table name and the second field name.

[0048] Furthermore, in the above-described device, when the acquisition module preprocesses the first data table name text, the first data table field name text, the second data table name text, and the second data table field name text respectively to generate the first data table name, the first field name, the second data table name, and the second field name, it is specifically configured to:

[0049] The first data table name text and the first data table field name text are preprocessed by removing stop words and extracting stems to generate the first data table name and the first field name; the second data table name text and the second data table field name text are preprocessed by removing stop words and extracting stems to generate the second data table name and the second field name.

[0050] Furthermore, in the above-mentioned device, the word vector model includes a word2vec model, a preset table vector algorithm and a preset field vector algorithm;

[0051] The first generating module is specifically configured to:

[0052] The word2vec model is used to split the first data table name and the first field name into multiple words, and the total number of words corresponding to all the words is determined; the word2vec model is used to convert the words into corresponding word vectors; all the word vectors and the total number of words are input into a preset table vector algorithm to generate a corresponding first table vector; the word vector corresponding to the first field name is input into a preset field vector algorithm to generate a corresponding first field vector; the second data table name and the second field name are vectorized based on the word vector model to generate a corresponding second table vector and a second field vector.

[0053] Furthermore, in the above device, the second generating module is specifically configured to:

[0054] The multilayer perceptron model is used to generate the probability corresponding to the table name blood relationship category of the first data table relative to the second data table based on the first table vector and the second table vector; the table name blood relationship category includes upstream blood relationship, downstream blood relationship and no blood relationship; if the probability corresponding to the upstream blood relationship is greater than the probability corresponding to the downstream blood relationship, and the probability corresponding to the upstream blood relationship is greater than the probability corresponding to no blood relationship, then the blood relationship corresponding to the table name between the first data table and the second data table is determined to be an upstream blood relationship with respect to the second data table; if the probability corresponding to the downstream blood relationship is greater than the probability corresponding to the upstream blood relationship, and the probability corresponding to the downstream blood relationship is greater than the probability corresponding to no blood relationship, then the blood relationship corresponding to the table name between the first data table and the second data table is determined to be a downstream blood relationship with respect to the second data table; if the probability corresponding to no blood relationship is greater than the probability corresponding to the upstream blood relationship, and the probability corresponding to no blood relationship is greater than the probability corresponding to the downstream blood relationship, then the blood relationship corresponding to the table name between the first data table and the second data table is determined to be no blood relationship with respect to the second data table.

[0055] Furthermore, in the above device, the preset similarity determination algorithm is a vector cosine similarity algorithm;

[0056] When the third generating module generates the blood relationship determination result corresponding to the field names between the first data table and the second data table based on the first field vector, the second field vector, and a preset similarity determination algorithm, it is specifically used to:

[0057] The first field vector and the second field vector are input into the vector cosine similarity algorithm to generate corresponding cosine similarity; if the cosine similarity is greater than a preset threshold, the result of determining the blood relationship corresponding to the field name between the first data table and the second data table is that a blood relationship exists; if the cosine similarity is less than or equal to the preset threshold, the result of determining the blood relationship corresponding to the field name between the first data table and the second data table is that no blood relationship exists.

[0058] Furthermore, the device as described above further comprises:

[0059] A training module is used to obtain training samples, wherein the training samples include: the actual blood relationship between each test data table, the data table name and the data table field name corresponding to the test data table; the training samples are input into a preset multi-layer perceptron model to train the preset multi-layer perceptron model; based on the test blood relationship between each test data table output by the preset multi-layer perceptron model and the actual blood relationship, it is determined whether the preset multi-layer perceptron model meets the preset convergence condition; if the preset multi-layer perceptron model meets the convergence condition, the preset multi-layer perceptron model that meets the convergence condition is determined as the preset multi-layer perceptron model trained to convergence.

[0060] A third aspect of the present application provides an electronic device, comprising: a memory and a processor;

[0061] The memory stores computer-executable instructions;

[0062] The processor executes the computer-executable instructions stored in the memory to implement the method for determining blood relationship in a data table as described in any one of the first aspects.

[0063] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method for determining blood relationships in a data table as described in any one of the first aspects.

[0064] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method for determining the blood relationship of a data table as described in any one of the first aspects.

[0065] The present application provides a method, apparatus, device, medium and product for determining blood relationship in a data table, the method comprising: obtaining a first intermediate parameter corresponding to a first data table to be determined blood relationship and a second intermediate parameter corresponding to a second data table; the first intermediate parameter comprises a first data table name and a first field name included in the first data table; the second intermediate parameter comprises a second data table name and a second field name included in the second data table; the first data table name, the first field name, the second data table name and the second field name are input into a word vector model trained to convergence, generating a corresponding first table vector and a first field vector, as well as a corresponding second table vector and a second field vector; the The first table vector is a vector corresponding to the name of the first data table, the first field vector is a vector corresponding to the name of the first field, the second table vector is a vector corresponding to the name of the second data table, and the second field vector is a vector corresponding to the name of the second field; the first table vector and the second table vector are input into the multi-layer perceptron model trained to convergence to generate a result of determining the blood relationship corresponding to the table names between the first data table and the second data table; if the result of determining the blood relationship corresponding to the table names between the first data table and the second data table is that there is a blood relationship, then based on the first field vector, the second field vector and the preset similarity determination algorithm, a result of determining the blood relationship corresponding to the field names between the first data table and the second data table is generated. The data table blood relationship determination method of the present application converts the intermediate parameters of the first data table and the second data table of the to-be-determined blood relationship into corresponding vectors through a word vector model, and compares the vectors corresponding to the first data table and the second data table through a multi-layer perceptron model and a preset similarity determination algorithm, thereby determining the blood relationship between the first data table and the second data table. Compared with the manual analysis method, it takes less time and improves the efficiency of determining the blood relationship of the data tables. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0067] Figure 1 This is a scene diagram that can implement the method for determining blood relationship in a data table according to an embodiment of the present application;

[0068] Figure 2 Schematic diagram of the process of determining blood relationship in the data table provided in this application Figure 1 ;

[0069] Figure 3 Schematic diagram of the process of determining blood relationship in the data table provided in this application Figure 2 ;

[0070] Figure 4 A schematic diagram of the word vector model training process for the method for determining blood relationship in a data table provided in this application;

[0071] Figure 5 A schematic diagram of the structure of the device for determining blood relationship in a data table provided in this application;

[0072] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application.

[0073] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0074] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0075] In the technical solutions of the embodiments of this application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0076] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0077] It should be noted that the disclosed methods, devices, equipment, media, and products for determining kinship relationships can be used in the field of financial technology or other related fields. They can also be used in any field other than financial technology or other related fields. The disclosed methods, devices, equipment, media, and products for determining kinship relationships are not limited to specific fields.

[0078] The technical solution of the present application is described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0079] In order to clearly understand the technical solution of this application, we first introduce the solution of the prior art in detail. A data table is a grid virtual table that stores data and represents a table of data in memory. A data table may have an association relationship with one or more other data tables, which is generally called a data table lineage relationship. Data table lineage relationships may have upstream relationships and downstream relationships. Traditional methods for determining upstream and downstream lineage relationships of data tables generally include the following two methods:

[0080] 1. Manual analysis: This method relies on manual analysis based on business knowledge and experience, by reviewing databases, source code, etc. to manually generate connections between tables. This method requires a lot of time and effort, and there is a possibility of omissions and errors.

[0081] 2. Database Log Analysis: This method analyzes database system log records to infer dependencies between tables. For related query statements, the execution path can be traced and all accessed tables can be recursively traversed during execution. However, log analysis technology is sometimes affected by database caching, human registration errors, missing system logs, and business logic optimization, resulting in inaccurate results and incomplete coverage of the entire data flow.

[0082] In general, these traditional methods have certain limitations in terms of accuracy and efficiency, and they cannot quickly cope with the ever-expanding and complex demands of data. Therefore, further optimization is needed to address the low efficiency of determining the lineage relationship of data tables.

[0083] Therefore, in order to address the low efficiency of determining data table kinship in existing technologies, the inventors discovered that, in order to solve this problem, a deep learning neural network can be used to obtain vector representations of the table names and field names of the data table. Using word vectorization technology, any type of data source can be converted into text format, and the words in the text (including table names and field names) are mapped into high-dimensional dense vectors, thereby achieving the unification and standardization of data sources, while also reflecting the dynamic changes in data kinship and improving the efficiency of kinship determination.

[0084] Specifically, obtain a first intermediate parameter corresponding to the first data table whose blood relationship is to be determined and a second intermediate parameter corresponding to the second data table. The first intermediate parameter includes the name of the first data table and the name of the first field included in the first data table. The second intermediate parameter includes the name of the second data table and the name of the second field included in the second data table. Input the first data table name, the first field name, the second data table name, and the second field name into a word vector model trained to convergence to generate a corresponding first table vector and a first field vector, as well as a corresponding second table vector and a second field vector. Input the first table vector and the second table vector into a multi-layer perceptron model trained to convergence to generate a blood relationship determination result corresponding to the table name between the first data table and the second data table. If the blood relationship determination result corresponding to the table name between the first data table and the second data table is that a blood relationship exists, then generate a blood relationship determination result corresponding to the field name between the first data table and the second data table based on the first field vector, the second field vector, and a preset similarity determination algorithm.

[0085] The method for determining the blood relationship of data tables in the present application converts the intermediate parameters of the first data table and the second data table whose blood relationship is to be determined into corresponding vectors through a word vector model, and compares the vectors corresponding to the first data table and the second data table through a multi-layer perceptron model and a preset similarity determination algorithm, thereby determining the blood relationship between the first data table and the second data table. Compared with the manual analysis method, it takes less time and improves the efficiency of determining the blood relationship of data tables.

[0086] Based on the above creative findings, the inventor proposed the technical solution of this application.

[0087] The following describes the application scenarios of the method for determining blood relationship in a data table provided in the embodiment of the present application. Figure 1 As shown, 1 is a first electronic device and 2 is a second electronic device. The network architecture of the application scenario corresponding to the data table blood relationship determination method provided in the embodiment of the present application includes: a first electronic device 1 and a second electronic device 2. The first electronic device 1 can be a blood relationship determination server, and the second electronic device 2 can be a database, etc. The second electronic device 2 stores the first data table and the second data table whose blood relationship is to be determined.

[0088] Exemplarily, when it is necessary to determine the blood relationship of a data table, the second electronic device 2 sends a first intermediate parameter corresponding to the first data table whose blood relationship is to be determined and a second intermediate parameter corresponding to the second data table to the first electronic device 1. The first intermediate parameter includes the name of the first data table and the first field name included in the first data table, and the second intermediate parameter includes the name of the second data table and the second field name included in the second data table. The first electronic device 1 inputs the first data table name, the first field name, the second data table name, and the second field name into a word vector model trained to convergence, and generates a corresponding first table vector and a first field vector, as well as a corresponding second table vector and a second field vector.

[0089] At the same time, the first electronic device 1 inputs the first table vector and the second table vector into the multi-layer perceptron model trained to convergence, and generates a result for determining the kinship relationship between the first data table and the second data table. If the result for determining the kinship relationship between the table names of the first data table and the second data table is that a kinship relationship exists, a result for determining the kinship relationship between the field names of the first data table and the second data table is generated based on the first field vector, the second field vector, and a preset similarity determination algorithm. If the result for determining the kinship relationship between the table names of the first data table and the second data table is that no kinship relationship exists, no similarity processing is performed between the first field vector and the second field vector, and it is determined that no kinship relationship exists between the first data table and the second data table. After determining the kinship relationship between the first data table and the second data table, the first electronic device 1 can send the kinship relationship determination result to the user terminal to inform the user of the kinship relationship between the first data table and the second data table.

[0090] The embodiments of the present application are introduced below with reference to the accompanying drawings.

[0091] Figure 2 Schematic diagram of the process of determining blood relationship in the data table provided in this application Figure 1 ,like Figure 2 As shown, in this embodiment, the execution subject of the embodiment of the present application is a data table blood relationship determination device, which can be integrated into an electronic device. The data table blood relationship determination method provided in this embodiment includes the following steps:

[0092] Step S101: Obtain a first intermediate parameter corresponding to a first data table whose blood relationship is to be determined and a second intermediate parameter corresponding to a second data table. The first intermediate parameter includes the name of the first data table and the name of the first field in the first data table. The second intermediate parameter includes the name of the second data table and the name of the second field in the second data table.

[0093] In this embodiment, each data table has a corresponding table name, such as a customer information table, a transaction data table, etc. Each data table generally has multiple field names. For example, a customer information table may have field names such as customer number, name, mobile phone number, and email address.

[0094] Step S102: Input the first data table name, first field name, second data table name, and second field name into the word vector model trained to convergence to generate corresponding first table vectors and first field vectors, as well as corresponding second table vectors and second field vectors. The first table vector is the vector corresponding to the first data table name, the first field vector is the vector corresponding to the first field name, the second table vector is the vector corresponding to the second data table name, and the second field vector is the vector corresponding to the second field name.

[0095] In the present embodiment, the word vector model and its principle word vectorization technology is to convert words into mathematical vector representations of fixed length so that computers can better understand and process natural language texts. The word2vec model commonly used in word vector technology is based on neural network training, and the vector representation of each word is obtained by learning text data. Specifically, word2vec uses two different models, namely the continuous bag of words model (abbreviated as CBOW in English, the full name in English is Continuous Bag of Words) and the skip-gram model. Both models are based on neural networks and generate high-quality vector representations for each word by learning contextual information from a large amount of text data. These vectors are usually represented in a low-dimensional space, and some semantic relationships can be retained, such as similar words in the vector space have similar vector representations.

[0096] The Continuous Bag of Words (CBOW) model is a neural network model used to train word vectors, predicting a target word based on the words in its context. Specifically, the CBOW model averages or sums the word representations in the context based on a window size (typically two or three words), then transforms these vectors into a probability distribution for the target word through a shallow neural network. During training, the CBOW model aims to maximize the probability of correctly predicting the target word, typically using a negative log-likelihood loss function for optimization.

[0097] The skip-gram model is a neural network model used to train word embeddings. Unlike the CBOW model, it predicts the contextual information surrounding the target word. Specifically, the skip-gram model takes a central word as input and attempts to predict the surrounding words that appear within a given window size. Therefore, rather than inputting context words and predicting the target word as the CBOW model does, the skip-gram model takes the central word as input and generates a probability distribution of surrounding words through a shallow neural network. During training, the skip-gram model aims to maximize the probability of correctly predicting surrounding words, typically using a negative log-likelihood function as the loss function for optimization.

[0098] Step S103 : Input the first table vector and the second table vector into the multi-layer perceptron model trained to convergence, and generate a result of determining the blood relationship between the first data table and the second data table corresponding to the table names.

[0099] In this embodiment, a multilayer perceptron model (MLP) is used. The MLP model typically consists of an input layer, several hidden layers, and an output layer. Each layer contains multiple neurons, each of which is connected to all neurons in the previous layer and has weight parameters for calculating the output of the neuron. The multilayer perceptron model can determine three types of kinship relationships between the table names of the first data table and the second data table based on the first table vector and the second table vector, such as upstream kinship, downstream kinship, and no kinship, as required.

[0100] Step S104: If the result of determining the blood relationship corresponding to the table names between the first data table and the second data table is that there is a blood relationship, a result of determining the blood relationship corresponding to the field names between the first data table and the second data table is generated based on the first field vector, the second field vector and a preset similarity determination algorithm.

[0101] In this embodiment, if the result of determining the blood relationship corresponding to the table name between the first data table and the second data table is that there is no blood relationship, the next step of the field name blood relationship determination process can be skipped, and it is determined that there is no blood relationship between the first data table and the second data table. The preset similarity determination algorithm can use an algorithm for calculating similarity, such as a cosine angle algorithm.

[0102] An embodiment of the present application provides a method for determining blood relationship in data tables. The intermediate parameters of a first data table and a second data table whose blood relationship is to be determined are converted into corresponding vectors through a word vector model, and the vectors corresponding to the first data table and the second data table are compared through a multi-layer perceptron model and a preset similarity determination algorithm, thereby determining the blood relationship between the first data table and the second data table. Compared with manual analysis methods, this method takes less time and improves the efficiency of determining the blood relationship in data tables.

[0103] Figure 3 Schematic diagram of the process of determining blood relationship in the data table provided in this application Figure 2 ,like Figure 3 As shown, the method for determining the blood relationship of a data table provided in this embodiment is a further refinement of the method for determining the blood relationship of a data table provided in the previous embodiment of this application. The method for determining the blood relationship of a data table provided in this embodiment includes the following steps.

[0104] Before processing, you can first train the CBOW and Skip-Gram word vector models. The training process is as follows:

[0105] The following is a detailed description of the training set inference process for the word vectorization technology (word2vec) using CBOW and Skip-Gram.

[0106] CBOW model:

[0107] The goal of the CBOW model is to predict the target word given the context words. Specifically, for a window with a fixed size of 2c, assuming the current position is t and the context word is

[0108] w t-c ,…,w t-1 ,w t+1 ,…,w t+c , the target word is w t The CBOW model uses the following formula to calculate the conditional probability:

[0109]

[0110] Among them, Z is the normalization factor, f(·) is a function that can be implemented by a neural network. Specifically, we use a neural network with a hidden layer size of d to represent the vector of the context word x t-c ,…,x t-1 ,x t+1 ,…,x t+c As input, it outputs the conditional probability of the target word. The structure of this neural network is as follows:

[0111]

[0112] u=Uh

[0113]

[0114] Among them, W i represents the weight matrix from the input layer to the hidden layer, U represents the weight matrix from the hidden layer to the output layer, and v j represents the vector representation of word j, |V| represents the size of the vocabulary, h and u represent the vector representation of the hidden layer and output layer respectively.

[0115] The training process of the CBOW model is to maximize the conditional probability. Suppose we have a training set D = (w (1) ,c (1) ),…,(w (|D|) ,c (|D|) ), where each sample (w (i) ,c (i) ) represents the target word w (i) and its context word c (i) Then, the loss function of the CBOW model can be defined as the average negative log likelihood (negative log

[0116] likelihood):

[0117]

[0118] During the training process, we maximize the loss function J through optimization algorithms such as stochastic gradient descent to obtain the vector representation of each word.

[0119] Skip-Gram Model:

[0120] The goal of the Skip-Gram model is to predict the surrounding context words given a target word. Specifically, for a window of fixed size 2c, assuming the current position is t and the target word is w t , the context word is w t-c ,…,w t-1 ,w t+1 ,…,w t+c The Skip-Gram model uses the following formula to calculate the conditional probability:

[0121]

[0122] Among them, P(w t+i |w t ) can be calculated by the following formula:

[0123]

[0124] Here we directly use the vector representation of the target word to predict surrounding words.

[0125] The training process of the Skip-Gram model is also the process of maximizing the conditional probability. Assume that we have a training set D = (w (1) ,c (1) ),…,(w (|D|) ,c (|D|) ), where each sample (w (i) ,c (i) ) represents the target word w (i) and its context word c (i) Then, the loss function of the Skip-Gram model can be defined as the average negative log-likelihood:

[0126]

[0127] Similarly, during the training process, we maximize the loss function J through optimization algorithms such as stochastic gradient descent to obtain the vector representation of each word.

[0128] In summary, in word2vec, we can train word embedding models using two approaches: CBOW and Skip-Gram. The CBOW model predicts a target word given context words, while the Skip-Gram model does the opposite, predicting context words given a target word. In both approaches, the goal is to maximize the conditional probability to obtain a vector representation for each word.

[0129] Step S201: Obtain a first data table and a second data table of blood relationships to be determined.

[0130] In this embodiment, the acquisition method may be to obtain from a database storing the first data table and the second data table.

[0131] Step S202: extracting data table name data and data table field data corresponding to the first data table and the second data table.

[0132] Step S203 : converting the data table name data and data table field data corresponding to the first data table into text format to generate a first data table name text and a first data table field name text.

[0133] Step S204 : converting the data table name data and data table field data corresponding to the second data table into text format to generate a second data table name text and a second data table field name text.

[0134] Step S205 , pre-processing the first data table name text, the first data table field name text, the second data table name text, and the second data table field name text respectively to generate the first data table name, the first field name, the second data table name, and the second field name.

[0135] In this embodiment, when the data source is vectorized through the word vectorization model, the data table name and data table field name of each data source need to be converted into text format, and interference in the text needs to be removed through preprocessing.

[0136] Optionally, in this embodiment, S205 may specifically be:

[0137] The first data table name text and the first data table field name text are pre-processed by removing stop words and extracting stems to generate the first data table name and the first field name.

[0138] The second data table name text and the second data table field name text are pre-processed by removing stop words and extracting stems to generate the second data table name and the second field name.

[0139] By performing stop word removal preprocessing and stem extraction preprocessing, the accurate first data table name and first field name, as well as the second data table name and second field name can be further extracted.

[0140] Step S206 , inputting the first data table name and the first field name into the word vector model trained to convergence to generate a corresponding first table vector and first field vector.

[0141] Optionally, in this embodiment, the word vector model includes a word2vec model, a preset table vector algorithm, and a preset field vector algorithm.

[0142] Then S206 may specifically be:

[0143] The word2vec model is used to split the first data table name and the first field name into multiple words, and the total number of words corresponding to all the words is determined.

[0144] The word2vec model is used to convert words into corresponding word vectors.

[0145] All word vectors and the total number of words are input into the preset table vector algorithm to generate the corresponding first table vector.

[0146] The word vector corresponding to the first field name is input into a preset field vector algorithm to generate a corresponding first field vector.

[0147] To obtain a vectorized representation of each table and field, you can use the preset table vectorization algorithm, as follows:

[0148]

[0149] Among them, Vec(T i ) represents the data table T i The vector representation, w represents the data table T i A word in the set consisting of the corresponding data table name and field name, T i Represents data table T i The corresponding data table name and field name constitute a set, and |·| represents the total number of words. In this embodiment, a simple average method is used to calculate the vector representation of the table.

[0150] The preset field vector algorithm is as follows:

[0151]

[0152] Among them, Vec(F i,j ) represents the vector representation of the j-th field name in the i data table, F i,j represents the name of the jth field in the i data table, and |·| represents the number of words in the field name.

[0153] For example, a data source contains multiple tables, such as customer information table, loan application form, repayment record table, etc., each table has more than 10 fields. Then the table name and field name of each table can be combined into one document, that is, the above data table T i A collection of corresponding data table names and field names.

[0154] The document content of the customer information table may be as follows:

[0155] "Customer Information Form Customer Number Name ID Number Mobile Number Email Address Occupation Gender Age"

[0156] At this point, we use the word2vec model to generate a vector representation for each word. Suppose we need to get a vector representation of the customer information table and the name field therein, we can use the following formula:

[0157]

[0158]

[0159] In this embodiment, a simple average method is used to calculate the vector representation of the table. For the name field, it is decomposed into two words, "surname" and "given name", and their vector representations are averaged to obtain the vector representation of the field.

[0160] Ultimately, the vector representation of each table and field can be stored in a database or other data warehouse for subsequent data analysis and mining. Furthermore, this vectorized representation method can be applied to any type of data source, allowing a variety of different types of data sources to be represented and processed uniformly.

[0161] At the same time, before S206 and S207, as Figure 4 As shown, you can also train a word2vec model by converting the table and field names of each data source into text format and then building a global corpus. Suppose there are multiple tabular data sources, each containing many tables and multiple fields. We combine the table and field names of each table into a document, then merge all the documents into a large corpus. This corpus is the dataset for training the word2vec model.

[0162] Next, during the training process, we first need to preprocess the text, such as removing stop words and performing stemming. We then use the word2vec model to train a vector representation for each word, generating a global word vector table. In this table, each table name and field name is assigned a unique vector representation. Table names are calculated using a word-weighted average, while field names are calculated using a single-word-weighted average.

[0163] Step S207 , inputting the second data table name and the second field name into the word vector model trained to convergence to generate corresponding second table vectors and second field vectors.

[0164] The steps of generating the second table vector and the second field vector are similar to S206 and will not be described in detail here. S207 and S206 may be executed simultaneously or before S206, which is not limited in this embodiment.

[0165] Step S208 : Input the first table vector and the second table vector into the multi-layer perceptron model trained to convergence, and generate a result of determining the blood relationship between the first data table and the second data table corresponding to the table names.

[0166] In this embodiment, the implementation of step 208 is similar to the implementation of step 103 in the previous embodiment, and will not be described in detail here.

[0167] Optionally, in this embodiment, S208 may specifically include:

[0168] A multilayer perceptron model is used to generate the probability of the first data table corresponding to the table name blood relationship category of the second data table based on the first table vector and the second table vector. The table name blood relationship categories include upstream blood relationship, downstream blood relationship, and no blood relationship.

[0169] If the probability corresponding to the upstream blood relationship is greater than the probability corresponding to the downstream blood relationship, and the probability corresponding to the upstream blood relationship is greater than the probability corresponding to no blood relationship, then the blood relationship corresponding to the table name between the first data table and the second data table is determined to be an upstream blood relationship between the first data table and the second data table.

[0170] If the probability corresponding to the downstream blood relationship is greater than the probability corresponding to the upstream blood relationship, and the probability corresponding to the downstream blood relationship is greater than the probability corresponding to no blood relationship, then the blood relationship corresponding to the table name between the first data table and the second data table is determined to be a downstream blood relationship between the first data table and the second data table.

[0171] If the probability corresponding to no blood relationship is greater than the probability corresponding to the upstream blood relationship, and the probability corresponding to no blood relationship is greater than the probability corresponding to the downstream blood relationship, then the blood relationship corresponding to the table name between the first data table and the second data table is determined to be that the first data table has no blood relationship with the second data table.

[0172] The upstream lineage relationship means that the first data table is the upstream table of the second data table, and the downstream lineage relationship means that the first data table is the downstream table of the second data table.

[0173] Optionally, in this embodiment, before S208, the multilayer perceptron model may be trained as follows:

[0174] A training sample is obtained, which includes: the actual relationship between the test data tables, the data table name and the data table field name corresponding to the test data table.

[0175] The training samples are input into the preset multilayer perceptron model to train the preset multilayer perceptron model.

[0176] Whether the preset multilayer perceptron model meets the preset convergence condition is determined according to the test kinship relationship and the actual kinship relationship between the test data tables output by the preset multilayer perceptron model.

[0177] If the preset multilayer perceptron model satisfies the convergence condition, the preset multilayer perceptron model that satisfies the convergence condition is determined as the preset multilayer perceptron model trained to convergence.

[0178] In this embodiment, the actual kinship relationship between the test data tables included in the training sample can be a directed graph of data tables. The upstream and downstream relationship of the data tables can be regarded as a directed graph relationship, in which each node represents a table. If table T i It is Table T j The upstream table of T i Pointing to T j To train the model, this directed graph and the test data table are converted into training samples.

[0179] Specifically, for each table T i , we can represent its vector x i As input features, all i A table with blood relationship, and a sampling part and T i Tables without blood relationship are aggregated into a set, denoted as N i Then, for N i Each table T in j , if T j It's T i The downstream table is labeled [1,0,0]. If T j It's T i The upstream table is labeled [0,1,0]. If T j It's T i There is no blood relationship, and the label is [0,0,1]. This gives a size of |N i | a set of three-class samples, where the feature of each sample is (x i , x j ), the label is a unique blood relationship represented by 0 or 1 [upstream table, downstream table, no relationship].

[0180] Next, we can use this training sample to train an MLP model to predict the kinship relationship between any two tables. An MLP model typically consists of an input layer, several hidden layers, and an output layer. Each layer contains multiple neurons, each of which is connected to all neurons in the previous layer and has weight parameters used to calculate its output.

[0181] In this embodiment, the vector representation of the data table can be used as the feature of the input layer, several hidden layers are set, and nonlinear mapping is performed using activation functions such as the sigmoid function or the ReLU function. Finally, a vector of length 3 is output, which respectively represents the probability that the table is an upstream table, the probability of a downstream table, and the probability of no relationship.

[0182] After training is complete, the trained MLP model can be used to infer the relationship between any two tables. Specifically, the vector representations of the two tables whose relationship needs to be determined are input, and the MLP model outputs the probability of the two tables being upstream, downstream, or unrelated. Based on the output, a threshold can be selected for judgment. For example, when the upstream probability is greater than the downstream probability and the probability of no relationship, the first table is considered the upstream table of the second. Conversely, when the downstream probability is greater than the upstream probability and the probability of no relationship, the first table is considered the downstream table of the second. If the probability of no relationship is the highest, the two tables can be considered unrelated.

[0183] Step S209: If the result of determining the blood relationship corresponding to the table names between the first data table and the second data table is that there is a blood relationship, a result of determining the blood relationship corresponding to the field names between the first data table and the second data table is generated based on the first field vector, the second field vector and a preset similarity determination algorithm.

[0184] In this embodiment, the implementation of step 209 is similar to the implementation of step 104 in the previous embodiment, and will not be described in detail here.

[0185] Optionally, in this embodiment, the preset similarity determination algorithm is a vector cosine similarity algorithm.

[0186] In S209, based on the first field vector, the second field vector, and the preset similarity determination algorithm, the result of determining the blood relationship corresponding to the field names between the first data table and the second data table may be generated as follows:

[0187] The first field vector and the second field vector are input into a vector cosine similarity algorithm to generate corresponding cosine similarities.

[0188] If the cosine similarity is greater than a preset threshold, a result of determining the blood relationship corresponding to the field names between the first data table and the second data table is that a blood relationship exists.

[0189] If the cosine similarity is less than or equal to a preset threshold, it is determined that the blood relationship corresponding to the field names between the first data table and the second data table does not exist.

[0190] Use the vector cosine similarity algorithm to calculate the similarity between vectors. The specific formula is as follows:

[0191]

[0192] Where x and y represent two vectors, x·y represents their dot product, and ||x|| and ||y|| represent their modulos. If the cosine similarity is greater than a preset threshold, the two vectors are considered similar, indicating that a blood relationship exists.

[0193] In order to further illustrate the method for determining blood relationship in a data table of this embodiment, a detailed description will be given below by way of examples.

[0194] This embodiment is specifically described using three tables of a bank customer transaction system, namely:

[0195] Customer Basic Information Form (Form 2)

[0196] Transaction Statement (Form 3)

[0197] Cross-border e-commerce consumption details table (Table 4)

[0198] The fields in these tables have been vectorized using word2vec technology. Each table and field has a unique 5-bit vector representation, as shown in Table 1:

[0199] Table 1 Vector table

[0200]

[0201] Table 2 Fields and vector tables in the "Customer Basic Information Table"

[0202]

[0203] Table 3 Fields and vectors in the "Transaction Statement Table"

[0204]

[0205] Table 4 Fields and vector tables in the "Cross-border e-commerce consumption details table"

[0206]

[0207] Assume that we need to infer the relationship between Table 1 (customer basic information table) and Table 2 (transaction statement table). We can input their respective vectors into the established MLP model to calculate the relationship between them.

[0208] Input: The vector representation of Table 1 is [0.245, -0.347, 0.123, 0.678, -0.456]

[0209] Input: The vector representation of Table 2 is [-0.678, 0.234, -0.570, 0.365, 0.987]

[0210] Calculations show that the probability of an upstream blood relationship is 0.83, the probability of a downstream blood relationship is 0.12, and the probability of no blood relationship is 0.05. Based on the principle of the softmax function, Table 1 can be considered the upstream blood relationship table of Table 2.

[0211] The softmax formula is as follows:

[0212]

[0213] Among them, f AB Indicates that table A is the score of the upstream table of table B, f BA Indicates that table A is the score of the downstream table of table B, f none The score indicating that there is no blood relationship between table A and table B.

[0214] Similarly, the relationship between Table 2 and Table 3 can also be processed as above, which will not be repeated here. The result is that there is no blood relationship.

[0215] Analysis of the blood relationship of table fields:

[0216] Still taking Table 1 and Table 2 as an example, calculate the field relationship between Table 1 "Customer Basic Information Table" and Table 2 "Transaction Statement Table" that has a blood relationship.

[0217] Now that the relationship between the table names of Table 1 and Table 2 has been determined, we need to find the field pairs that have a relationship between the two tables.

[0218] The vector cosine angle calculation formula can be used to calculate the similarity between the fields in each table, and to find field pairs with an angle greater than the preset threshold of 0.8. These field pairs can be considered to be related.

[0219] Input: Field vector representations of Table 1 "Customer Basic Information Table" and Table 2 "Transaction Statement Table"

[0220] Calculate the cosine angle between each field vector and get the following table:

[0221] Table 5 Angle results

[0222]

[0223] Based on the size of the cosine angle, we can identify field pairs that meet a threshold greater than 0.8. These field pairs are considered to have a blood relationship. In this example, the angle between the customer number and customer code is 0.81, and the angle between the customer name and the counterparty account is 0.86. Therefore, these two pairs of fields are considered to have a blood relationship.

[0224] Since there is no blood relationship between Table 2 and Table 3, there is no need to calculate the cosine of the vector angle between the fields.

[0225] The data table kinship determination method of this embodiment uses the word vectorization technology (word2vec) method to obtain vector expressions of table names and field names by deep learning neural network vectorization of data table fields and data table names. Using word vectorization technology (word2vec), any type of data source can be converted into text format, and the words in the text (including table names and field names) are mapped into high-dimensional dense vectors. This can achieve the unification and standardization of data sources, thereby facilitating subsequent data analysis and mining, reflecting the dynamic changes of data kinship relationships, and improving the storage and query efficiency of kinship relationships.

[0226] At the same time, it has the following advantages:

[0227] 1. You can convert any type of data source table name and field name into text format, and then use word vectorization technology to map the words in the text (including table names and field names) into high-dimensional dense vectors, thereby achieving the unification and standardization of data source identification.

[0228] 2. The window mechanism and negative sampling mechanism of word vectorization technology can be used to capture contextual and semantic information in data transformation logic, thereby extracting implicit relationships between data tables and fields.

[0229] 3. The online learning capabilities of word vectorization technology can be used to update the vector representations of data tables and fields in real time, thereby reflecting the dynamic changes in data lineage relationships.

[0230] 4. Distance measurement and similarity calculation in vector space can be used to quickly query and retrieve the blood relationship between data tables and fields, thereby improving the storage and query efficiency of data blood relationship.

[0231] Figure 5 The schematic diagram of the structure of the device for determining blood relationship of the data table provided in this application is as follows: Figure 5 As shown, in this embodiment, the data table blood relationship determination device 300 can be set in an electronic device, such as a terminal device, and the data table blood relationship determination device 300 includes:

[0232] Acquisition module 301 is configured to acquire a first intermediate parameter corresponding to a first data table for determining a blood relationship and a second intermediate parameter corresponding to a second data table. The first intermediate parameter includes the name of the first data table and the name of a first field in the first data table. The second intermediate parameter includes the name of the second data table and the name of a second field in the second data table.

[0233] The first generation module 302 is configured to input the first data table name, the first field name, the second data table name, and the second field name into the word vector model trained to convergence, and generate corresponding first table vectors and first field vectors, as well as corresponding second table vectors and second field vectors. The first table vector is a vector corresponding to the first data table name, the first field vector is a vector corresponding to the first field name, the second table vector is a vector corresponding to the second data table name, and the second field vector is a vector corresponding to the second field name.

[0234] The second generating module 303 is configured to input the first table vector and the second table vector into the multi-layer perceptron model trained to convergence, and generate a result of determining the blood relationship between the first data table and the second data table corresponding to the table names.

[0235] The third generation module 304 is used to generate a blood relationship determination result corresponding to the field name between the first data table and the second data table based on the first field vector, the second field vector and a preset similarity determination algorithm if the blood relationship determination result corresponding to the table name between the first data table and the second data table is that a blood relationship exists.

[0236] The data table blood relationship determination device provided in this embodiment can be executed Figure 2 The technical solution of the method embodiment shown in the figure has the same implementation principle and technical effect as Figure 2 The method embodiments shown are similar and will not be described in detail here.

[0237] The data table blood relationship determination device provided in this application is a further refinement of the data table blood relationship determination device provided in the previous embodiment. The data table blood relationship determination device 300 includes:

[0238] Optionally, in this embodiment, the acquisition module 301 is specifically configured to:

[0239] Obtain the first data table and the second data table whose blood relationship is to be determined. Extract the data table name data and data table field data corresponding to the first data table and the second data table. Convert the data table name data and data table field data corresponding to the first data table into text format to generate the first data table name text and the first data table field name text. Convert the data table name data and data table field data corresponding to the second data table into text format to generate the second data table name text and the second data table field name text. Preprocess the first data table name text, the first data table field name text, the second data table name text, and the second data table field name text respectively to generate the first data table name, the first field name, the second data table name, and the second field name.

[0240] Optionally, in this embodiment, when the acquisition module 301 preprocesses the first data table name text, the first data table field name text, the second data table name text, and the second data table field name text respectively to generate the first data table name, the first field name, the second data table name, and the second field name, it is specifically configured to:

[0241] The first data table name text and the first data table field name text are pre-processed by removing stop words and extracting stems to generate the first data table name and the first field name. The second data table name text and the second data table field name text are pre-processed by removing stop words and extracting stems to generate the second data table name and the second field name.

[0242] Optionally, in this embodiment, the word vector model includes a word2vec model, a preset table vector algorithm, and a preset field vector algorithm.

[0243] The first generating module 302 is specifically configured to:

[0244] Use the word2vec model to split the first data table name and the first field name into multiple words, and determine the total number of words corresponding to all the words. Use the word2vec model to convert the words into corresponding word vectors. Input all word vectors and the total number of words into a preset table vector algorithm to generate a corresponding first table vector. Input the word vector corresponding to the first field name into a preset field vector algorithm to generate a corresponding first field vector. Vectorize the second data table name and the second field name based on the word vector model to generate a corresponding second table vector and second field vector.

[0245] Optionally, in this embodiment, the second generating module 303 is specifically configured to:

[0246] A multilayer perceptron model is used to generate the probabilities corresponding to the table name blood relationship categories of the first data table relative to the second data table based on the first table vector and the second table vector. Table name blood relationship categories include upstream blood relationship, downstream blood relationship, and no blood relationship. If the probability corresponding to the upstream blood relationship is greater than the probability corresponding to the downstream blood relationship, and the probability corresponding to the upstream blood relationship is greater than the probability corresponding to the no blood relationship, then the blood relationship corresponding to the table name between the first data table and the second data table is determined to be an upstream blood relationship with respect to the second data table. If the probability corresponding to the downstream blood relationship is greater than the probability corresponding to the upstream blood relationship, and the probability corresponding to the downstream blood relationship is greater than the probability corresponding to the no blood relationship, then the blood relationship corresponding to the table name between the first data table and the second data table is determined to be a downstream blood relationship with respect to the second data table. If the probability corresponding to the no blood relationship is greater than the probability corresponding to the upstream blood relationship, and the probability corresponding to the no blood relationship is greater than the probability corresponding to the downstream blood relationship, then the blood relationship corresponding to the table name between the first data table and the second data table is determined to be no blood relationship with respect to the second data table.

[0247] Optionally, in this embodiment, the preset similarity determination algorithm is a vector cosine similarity algorithm.

[0248] When the third generating module 304 generates the result of determining the blood relationship corresponding to the field names between the first data table and the second data table based on the first field vector, the second field vector, and a preset similarity determination algorithm, it is specifically configured to:

[0249] The first field vector and the second field vector are input into a vector cosine similarity algorithm to generate corresponding cosine similarities. If the cosine similarity is greater than a preset threshold, the result of determining the blood relationship corresponding to the field name between the first data table and the second data table is that a blood relationship exists. If the cosine similarity is less than or equal to the preset threshold, the result of determining the blood relationship corresponding to the field name between the first data table and the second data table is that no blood relationship exists.

[0250] Optionally, in this embodiment, the data table blood relationship determination device 300 further includes:

[0251] The training module is configured to obtain training samples, including the actual kinship relationships between the test data tables, the data table names corresponding to the test data tables, and the data table field names. The training samples are input into a preset multilayer perceptron model to train the preset multilayer perceptron model. A determination is made as to whether the preset multilayer perceptron model satisfies a preset convergence condition based on the test kinship relationships between the test data tables output by the preset multilayer perceptron model and the actual kinship relationships. If the preset multilayer perceptron model satisfies the convergence condition, the preset multilayer perceptron model that satisfies the convergence condition is determined as the preset multilayer perceptron model trained to convergence.

[0252] The data table blood relationship determination device provided in this embodiment can be executed Figure 2-Figure 4 The technical solution of the method embodiment shown in the figure has the same implementation principle and technical effect as Figure 2-Figure 4 The method embodiments shown are similar and will not be described in detail here.

[0253] According to an embodiment of the present application, the present application also provides an electronic device, a computer-readable storage medium, and a computer program product.

[0254] like Figure 6 As shown, Figure 6 is a schematic diagram of the structure of the electronic device provided by the present application. The electronic device is intended to be various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application as described and / or claimed herein.

[0255] like Figure 6 As shown, the electronic device includes: a processor 401 and a memory 402. The various components are connected to each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the electronic device.

[0256] Memory 402 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor, causing the at least one processor to perform the method for determining blood relationships in a data table provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the method for determining blood relationships in a data table provided in this application.

[0257] The memory 402 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the method for determining blood relationship in a data table in the embodiment of the present application (for example, the attached Figure 5 The processor 401 executes the non-transient software programs, instructions, and modules stored in the memory 402 to execute various functional applications and data processing of the electronic device, thereby implementing the data table blood relationship determination method in the above method embodiment.

[0258] At the same time, this embodiment also provides a computer product. When the instructions in the computer product are executed by the processor of an electronic device, the electronic device can execute the data table blood relationship determination method of the above embodiment.

[0259] Those skilled in the art will readily conceive of other implementations of the embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the embodiments of the present application, which follow the general principles of the embodiments of the present application and include common knowledge or customary technical means in the art not disclosed in the embodiments of the present application.

[0260] It should be understood that the embodiments of the present application are not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the embodiments of the present application is limited only by the appended claims.

Claims

1. A method for determining blood relationship in a data table, characterized in that: include: Obtaining a first intermediate parameter corresponding to the first data table for determining the blood relationship and a second intermediate parameter corresponding to the second data table; The first intermediate parameter includes a first data table name and a first field name included in the first data table; the second intermediate parameter includes a second data table name and a second field name included in the second data table; Input the first data table name, the first field name, the second data table name, and the second field name into the word vector model trained to convergence to generate the corresponding first table vector and the first field vector, as well as the corresponding second table vector and the second field vector; the first table vector is the vector corresponding to the first data table name, the first field vector is the vector corresponding to the first field name, the second table vector is the vector corresponding to the second data table name, and the second field vector is the vector corresponding to the second field name; Inputting the first table vector and the second table vector into a multi-layer perceptron model trained to convergence, generating a result of determining the blood relationship between the first data table and the second data table corresponding to the table name; If the result of determining the blood relationship corresponding to the table names between the first data table and the second data table is that there is a blood relationship, then a result of determining the blood relationship corresponding to the field names between the first data table and the second data table is generated based on the first field vector, the second field vector and the preset similarity determination algorithm.

2. The method according to claim 1, characterized in that The step of obtaining a first intermediate parameter corresponding to the first data table and a second intermediate parameter corresponding to the second data table of the blood relationship to be determined includes: Obtaining a first data table and a second data table of the blood relationship to be determined; Extracting data table name data and data table field data corresponding to the first data table and the second data table; Convert the data table name data and data table field data corresponding to the first data table into text format to generate first data table name text and first data table field name text; Convert the data table name data and data table field data corresponding to the second data table into text format to generate a second data table name text and a second data table field name text; The first data table name text, the first data table field name text, the second data table name text and the second data table field name text are preprocessed respectively to generate the first data table name, the first field name, the second data table name and the second field name.

3. The method according to claim 1, characterized in that The word vector model includes a word2vec model, a preset table vector algorithm and a preset field vector algorithm; The step of inputting the first data table name, the first field name, the second data table name, and the second field name into a word vector model trained to convergence to generate a corresponding first table vector and a first field vector, as well as a corresponding second table vector and a second field vector includes: Split the first data table name and the first field name into multiple words using a word2vec model, and determine the total number of words corresponding to all the words; Use word2vec model to convert the words into corresponding word vectors; Inputting all the word vectors and the total number of words into a preset table vector algorithm to generate a corresponding first table vector; Inputting the word vector corresponding to the first field name into a preset field vector algorithm to generate a corresponding first field vector; The second data table name and the second field name are vectorized based on the word vector model to generate corresponding second table vectors and second field vectors.

4. The method according to claim 1, wherein The step of inputting the first table vector and the second table vector into a multi-layer perceptron model trained to convergence to generate a result of determining the blood relationship between the first data table and the second data table corresponding to the table names includes: The multilayer perceptron model is used to generate a probability corresponding to a table name blood relationship category of the first data table relative to the second data table based on the first table vector and the second table vector; the table name blood relationship category includes upstream blood relationship, downstream blood relationship, and no blood relationship; If the probability corresponding to the upstream blood relationship is greater than the probability corresponding to the downstream blood relationship, and the probability corresponding to the upstream blood relationship is greater than the probability corresponding to no blood relationship, then the blood relationship corresponding to the table names between the first data table and the second data table is determined to be an upstream blood relationship between the first data table and the second data table; If the probability corresponding to the downstream blood relationship is greater than the probability corresponding to the upstream blood relationship, and the probability corresponding to the downstream blood relationship is greater than the probability corresponding to no blood relationship, then the blood relationship corresponding to the table names between the first data table and the second data table is determined to be a downstream blood relationship between the first data table and the second data table; If the probability corresponding to no blood relationship is greater than the probability corresponding to the upstream blood relationship, and the probability corresponding to no blood relationship is greater than the probability corresponding to the downstream blood relationship, then the blood relationship corresponding to the table name between the first data table and the second data table is determined to be that the first data table has no blood relationship with the second data table.

5. The method according to claim 1, wherein The preset similarity determination algorithm is a vector cosine similarity algorithm; The generating, based on the first field vector, the second field vector, and a preset similarity determination algorithm, a result of determining the blood relationship corresponding to the field names between the first data table and the second data table includes: Inputting the first field vector and the second field vector into the vector cosine similarity algorithm to generate corresponding cosine similarities; If the cosine similarity is greater than a preset threshold, determining that the blood relationship corresponding to the field names between the first data table and the second data table exists is a blood relationship; If the cosine similarity is less than or equal to a preset threshold, it is determined that the blood relationship corresponding to the field names between the first data table and the second data table does not exist.

6. The method according to any one of claims 1 to 5, characterized in that Before inputting the first table vector and the second table vector into the multi-layer perceptron model trained to convergence and generating the result of determining the blood relationship corresponding to the table names between the first data table and the second data table, the method further includes: Obtaining a training sample, wherein the training sample includes: the actual blood relationship between each test data table, the data table name and data table field name corresponding to the test data table; Inputting the training samples into a preset multi-layer perceptron model to train the preset multi-layer perceptron model; Determining whether the preset multilayer perceptron model satisfies a preset convergence condition based on the test kinship relationship between the test data tables output by the preset multilayer perceptron model and the actual kinship relationship; If the preset multilayer perceptron model satisfies the convergence condition, the preset multilayer perceptron model that satisfies the convergence condition is determined as the preset multilayer perceptron model trained to convergence.

7. A device for determining blood relationship in a data table, characterized in that: include: An acquisition module, configured to acquire a first intermediate parameter corresponding to a first data table and a second intermediate parameter corresponding to a second data table for determining blood relationship; The first intermediate parameter includes a first data table name and a first field name included in the first data table; the second intermediate parameter includes a second data table name and a second field name included in the second data table; A first generation module is configured to input the first data table name, the first field name, the second data table name, and the second field name into a word vector model trained to convergence, and generate a corresponding first table vector and first field vector, as well as a corresponding second table vector and second field vector; the first table vector is a vector corresponding to the first data table name, the first field vector is a vector corresponding to the first field name, the second table vector is a vector corresponding to the second data table name, and the second field vector is a vector corresponding to the second field name; A second generating module is configured to input the first table vector and the second table vector into a multi-layer perceptron model trained to convergence, and generate a result of determining the blood relationship between the first data table and the second data table corresponding to the table names; The third generation module is used to generate a blood relationship determination result corresponding to the field name between the first data table and the second data table based on the first field vector, the second field vector and a preset similarity determination algorithm if the blood relationship determination result corresponding to the table name between the first data table and the second data table is that there is a blood relationship.

8. An electronic device, characterized in that: include: memory and processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method for determining blood relationship in a data table according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method for determining blood relationship in a data table according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for determining blood relationship in a data table according to any one of claims 1 to 6 is implemented.

Images