Method and device for constructing data modeling association, electronic equipment and storage medium

Abstract

The present disclosure relates to a method and device for constructing data modeling association, electronic equipment and storage medium, the method comprising: scanning a plurality of table files in a plurality of databases contained in a target system to obtain library table data corresponding to each of the plurality of table files, the library table data at least including business data, then obtaining table data of each table file and a plurality of field data contained in the table data according to the business data of each table file, processing the plurality of fields, and establishing an association relationship between the plurality of table files. When the relevant documents, logs, specifications and other information are missing or imperfect, the present disclosure obtains the library table data of each table file according to the table files contained in all databases in the target system, then directly processes the library table data to obtain the distance between the field vectors corresponding to any two field data pairs, and establishes the association relationship between the table files, thereby quickly helping data inventory understanding.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and particularly relates to a method and device for constructing data modeling association, electronic equipment and storage medium. BACKGROUND

[0002] For data inventory, manual recording or data catalog recording system is generally adopted to record data inventory information, but data is much, business is complex and data is in continuous change, which makes it difficult for data inventory work to withstand, and it is urgent to improve the dynamic ability of business big data inventory understanding.

[0003] In related technologies, the metadata construction scheme of business data is established on the basis of standard documents, database compliance paradigm, database log, and function system log. However, in the early stage of data center construction, there are certain problems in the data inventory related information elements, and even missing conditions, for example: the database design paradigm compliance is incomplete, the association in the library table (table file) is missing, the database query log is not opened due to performance reasons, the association between library tables is missing, the business system log does not reasonably record the association between data and functions, and the association between data and functions is missing, thereby causing difficulty in data inventory. SUMMARY

[0004] The present disclosure provides a method and device for constructing data modeling association, electronic equipment and storage medium to at least solve the problem of difficulty in data inventory caused by missing data association in related technologies. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a method for constructing data modeling association is provided, comprising:

[0006] scanning a plurality of table files in a plurality of databases contained in a target system to obtain library table data corresponding to each of the plurality of table files, wherein the library table data at least includes business data;

[0007] According to the business data of each table file, obtaining table data of each table file and a plurality of field data contained in the table data;

[0008] vectorizing the plurality of field data to obtain field vectors corresponding to each of the plurality of field data;

[0009] According to the field vectors corresponding to each of the plurality of field data, obtaining the distance between the field vectors corresponding to any two field data;

[0010] According to the distance between the field vectors corresponding to any two field data in the plurality of field data and the plurality of field data contained in the table data of each table file, establishing an association relationship between the plurality of table files.

[0011] Optionally, the library table data further comprises table description, and the table description of each table file comprises table structure of the table file and field type of each field data contained in the table file.

[0012] Before the vectorization of the plurality of field data to obtain the field vectors corresponding to the plurality of field data respectively, the method further comprises:

[0013] According to the field type of each field data contained in each table file, target field data is obtained from the plurality of field data, and the target field data at least comprises field data of the relationship type;

[0014] The vectorization of the plurality of field data to obtain the field vectors corresponding to the plurality of field data respectively comprises:

[0015] The vectorization of the plurality of field data of the relationship type in the target field data to obtain the field vectors corresponding to the plurality of field data respectively.

[0016] Optionally, according to the distance between the field vectors corresponding to any two field data in the plurality of field data and the plurality of field data contained in the table data of each table file, an association relationship between the plurality of table files is established, comprising:

[0017] According to the distance between the field vectors corresponding to any two field data in the plurality of field data, an association relationship between the plurality of field data is established to obtain a field knowledge graph, and the field knowledge graph comprises a plurality of association relationship groups, and each association relationship group is composed of a plurality of field data of the established association relationship;

[0018] According to the field knowledge graph and the plurality of field data contained in the table data of each table file, an association relationship between the plurality of table files is established to obtain a table file knowledge graph.

[0019] Optionally, the target field data further comprises field data of the time sequence type;

[0020] After the association relationship between the plurality of table files is established according to the field knowledge graph and the plurality of field data contained in the table data of each table file to obtain the table file knowledge graph, the method further comprises:

[0021] According to the field data of the relationship type and the field data of the time sequence type in each table file, a time sequence description of the association relationship between the field data in each table file is performed to obtain a table file knowledge graph with time sequence.

[0022] Optionally, after the association between the field data in each table file is described in time sequence according to the field data of the relationship type and the field data of the time sequence type in each table file, the table file knowledge graph with time sequence is obtained, the method further comprises:

[0023] generating a system interface for data inventory management according to the table file knowledge graph with time sequence;

[0024] generating a business theme according to the constraint condition of the system interface and the table file knowledge graph with time sequence.

[0025] Optionally, the plurality of field data is vectorized to obtain a field vector corresponding to each of the plurality of field data, including:

[0026] determining the number of elements expected to be supported by the filter and the filter bit array value according to the plurality of table files;

[0027] determining the number K of hash functions according to the number of elements expected to be supported by the filter, the false positive probability and the filter bit array value;

[0028] for each field data, mapping the field data into K points in a bit array through K hash functions, and replacing 0 in the K points in the bit array with 1, the positions of the K points obtained by different field data are not completely the same;

[0029] obtaining a field vector corresponding to each field data according to the bit array obtained by each field data.

[0030] Optionally, according to the field vector corresponding to each of the plurality of field data, the distance between the field vectors corresponding to any two field data is obtained, including:

[0031] calculating the distance between the field vectors corresponding to any two field data in the field vectors corresponding to the plurality of field data, wherein the distance calculation method includes at least one of the following: Euclidean distance, Manhattan distance, Chebyshev distance, Hamming distance and cosine distance.

[0032] According to a second aspect of the embodiments of the present disclosure, an apparatus for constructing data modeling association is provided, including:

[0033] a scanning module configured to scan a plurality of table files in a plurality of databases contained in a target system to obtain library table data corresponding to each of the plurality of table files, the library table data at least including business data;

[0034] an obtaining module configured to obtain table data of each table file and a plurality of field data contained in the table data according to the business data of each table file;

[0035] The first obtaining module is configured to vectorize the multiple field data to obtain field vectors corresponding to the multiple field data respectively.

[0036] The second obtaining module is configured to obtain distances between field vectors corresponding to any two field data according to the field vectors corresponding to the multiple field data respectively.

[0037] The establishing module is configured to establish an association relationship between the multiple table files according to the distances between the field vectors corresponding to any two field data in the multiple field data and the multiple field data contained in the table data of each table file.

[0038] Optionally, the library table data further includes table building descriptions, and each table building description of each table file includes a table structure of the table file and a field type of each field data contained in the table file.

[0039] Before the first obtaining module, the apparatus further includes:

[0040] The third obtaining module is configured to obtain target field data from the multiple field data according to the field type of each field data contained in each table file, and the target field data at least includes relationship type field data.

[0041] The first obtaining module includes:

[0042] The first obtaining submodule is configured to vectorize multiple relationship type field data in the target field data to obtain field vectors corresponding to the multiple field data respectively.

[0043] Optionally, the establishing module includes:

[0044] The first establishing submodule is configured to establish an association relationship between the multiple field data according to the distances between the field vectors corresponding to any two field data in the multiple field data, to obtain a field knowledge graph, and the field knowledge graph includes multiple association relationship groups, and each association relationship group is composed of multiple field data with which the association relationship is established.

[0045] The second establishing submodule is configured to establish an association relationship between the multiple table files according to the field knowledge graph and the multiple field data contained in the table data of each table file, to obtain a table file knowledge graph.

[0046] Optionally, the target field data further includes time sequence type field data.

[0047] After the second establishing submodule, the apparatus further includes:

[0048] The fourth obtaining module is configured to perform time sequence description on the association relationship between the field data in each table file according to the field data of the relationship type and the field data of the time sequence type in each table file, and obtain a table file knowledge graph with time sequence.

[0049] Optionally, after the fourth obtaining module, the apparatus further comprises:

[0050] The first generating module is configured to generate a system interface for data inventory management according to the table file knowledge graph with time sequence.

[0051] The second generating module is configured to generate a business theme according to the constraint condition of the system interface and the table file knowledge graph with time sequence.

[0052] Optionally, the first obtaining module comprises:

[0053] The first determining submodule is configured to determine the number of elements expected to be supported by the filter and the filter bit array value according to the plurality of table files.

[0054] The second determining submodule is configured to determine the number K of hash functions according to the number of elements expected to be supported by the filter, the false positive probability and the filter bit array value.

[0055] The mapping submodule is configured to map each field data into K points in a bit array through K hash functions, and replace 0 in the K points in the bit array with 1, and the positions of the K points obtained by different field data are not completely the same.

[0056] The obtaining submodule is configured to obtain a field vector corresponding to each field data according to the bit array obtained by each field data.

[0057] Optionally, the second obtaining module comprises:

[0058] The calculating submodule is configured to calculate the distance between the field vectors corresponding to any two field data in the field vectors corresponding to the plurality of field data respectively, wherein the distance calculation method comprises at least one of the following: Euclidean distance, Manhattan distance, Chebyshev distance, Hamming distance and cosine distance.

[0059] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, comprising:

[0060] a processor;

[0061] a memory for storing instructions executable by the processor;

[0062] wherein the processor is configured to execute the instructions to implement the method for constructing data modeling association according to the first aspect.

[0063] According to a fourth aspect of the embodiments of the present disclosure, a storage medium is provided, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device can perform the method for constructing data modeling association as described in the first aspect.

[0064] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, the computer program product comprises readable program codes, when the readable program codes are executed by a processor of an electronic device, the electronic device can perform the method for constructing data modeling association as described in the first aspect.

[0065] The embodiments of the present disclosure provide at least the following beneficial effects:

[0066] The present disclosure scans a plurality of table files in a plurality of databases contained in a target system to obtain library table data corresponding to each of the plurality of table files, the library table data at least including business data, then according to the business data of each table file, obtains table data of each table file and a plurality of field data contained in the table data, and then vectorizes the plurality of field data to obtain field vectors corresponding to each of the plurality of field data, and then according to the field vectors corresponding to each of the plurality of field data, obtains distances between field vectors corresponding to any two field data, and finally according to the distances between the field vectors corresponding to any two field data in the plurality of field data and the plurality of field data contained in the table data of each table file, establishes an association relationship between the plurality of table files. When the relevant documents, logs, specifications and other information are missing or imperfect, the present disclosure obtains the library table data of each table file according to the table files contained in all databases in the target system, then directly processes the library table data to obtain the distances between field vectors corresponding to any two field data, and establishes the association relationship between the table files, thereby quickly helping data inventory understanding.

[0067] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0068] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure, and do not constitute an improper limitation on the present disclosure.

[0069] Figure 1 is a step flowchart of a method for constructing data modeling association according to an exemplary embodiment;

[0070] Figure 2 is a flowchart of a method for vectorizing field data according to an exemplary embodiment;

[0071] Figure 3 is a flow chart of a method for building data modeling association according to an exemplary embodiment;

[0072] Figure 4 is a block diagram of an apparatus for building data modeling association according to an exemplary embodiment;

[0073] Figure 5 is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0074] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings.

[0075] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0076] Figure 1 is a step flow chart of a method for building data modeling association according to an exemplary embodiment, as shown in Figure 1 the method for building data modeling association comprises the following steps:

[0077] In step S11, a plurality of table files in a plurality of databases included in a target system are scanned to obtain a plurality of library table data respectively corresponding to the plurality of table files, the library table data at least including business data.

[0078] The target system is a system that currently needs to be data inventoried. The target system can include multiple systems. For example, when the target system is a plurality of business management systems of a school, the target system can include an examination management system, an enrollment management system, and a student management system. Each system can include a plurality of databases, each database containing a plurality of table files, each table file recording a plurality of data information according to a predetermined format, and each table file corresponding to a library table data. By scanning the plurality of table files in the plurality of databases included in the target system, all library table data corresponding to the table files included in the target system can be obtained, wherein the library table data can include business data, and the business data is all data recorded by the table file.

[0079] In step S12, according to the service data of each table file, the table data of each table file and the plurality of field data contained in the table data are obtained.

[0080] The service data of each table file contains the table data and the corresponding specific parameters, the table data represents the data items recorded by the corresponding table file, for example, if the table file is a student basic information table, the service data is "serial number, certificate type, certificate number, name, gender, comprehensive evaluation", the table data can be obtained from the service data of each table file, and the plurality of field data contained in the table data can be obtained from the table data, each field data is each data item, for example, one of the field data can be "certificate type" or "comprehensive evaluation", the service data further includes the specific parameters corresponding to each data item, for example, one of the specific parameters corresponding to the certificate number is "362227199110000000".

[0081] In step S13, the plurality of field data are vectorized to obtain the field vectors corresponding to the plurality of field data respectively.

[0082] By vectorizing the plurality of field data obtained, the field vectors corresponding to the plurality of field data respectively are obtained, which can facilitate finding the association relationship between the field data.

[0083] In step S14, according to the field vectors corresponding to the plurality of field data respectively, the distance between the field vectors corresponding to any two field data is obtained.

[0084] By processing the field vectors corresponding to the plurality of field data respectively, specifically, the field vectors corresponding to any two field data are combined, and the distance between the field vectors corresponding to any two field data is calculated, so as to facilitate determining the association relationship between any two field data by quantifying data.

[0085] In step S15, according to the distance between the field vectors corresponding to any two field data in the plurality of field data and the plurality of field data contained in the table data of each table file, the association relationship between the plurality of table files is established.

[0086] According to the distance between the field vectors corresponding to any two field data, if the distance between the field vectors corresponding to any two field data is greater than a preset distance threshold, the data level recognition scanning of the field data can be performed, wherein the preset distance threshold can be 0.5, and the data level recognition scanning can be value range boundary scanning and coincidence degree scanning, thereby establishing the association relationship for the field data capable of establishing the association relationship, and obtaining the relationship between each field data and the table file according to the plurality of field data contained by the table data of each table file, thereby further establishing the association relationship between the plurality of table files, wherein the association relationship between the plurality of table files includes the intra-table file association relationship and the inter-table file association relationship.

[0087] In the business data inventory work, in the case of missing specification documents, log files and library table creation not following the paradigm. According to the table file contained by all the databases in the target system, the library table data of each table file is obtained, then the library table data is directly processed to obtain the distance between the field vectors corresponding to any two field data, that is, the association relationship between the table files can be established, thereby quickly helping the data inventory to understand.

[0088] On the basis of the above technical solutions, since the number of table files obtained from the target system is huge, the number of field data obtained is also large, in order to improve the efficiency of establishing the association relationship between the field data, a part of the field data capable of constructing the association relationship can be selected for further processing according to the field type of the field data, specifically:

[0089] The library table data further includes table creation description, and the table creation description of each table file includes the table structure of the table file and the field type of each field data contained by the table file.

[0090] Before step S13, the method further includes:

[0091] According to the field type of each field data contained by each table file, target field data is obtained from the plurality of field data, and the target field data at least includes the field data of the relationship type.

[0092] According to the field type of each field data contained in each table file, the selecting part can construct the field data of the part of the relationship as target field data. Specifically, a preset field type can be set to filter the field data capable of constructing the relationship. If the field type of the field data is the same as one of the preset field types, it is determined that the field data is the field data capable of constructing the relationship. The preset field types can be numerical type, floating point type, binary type, etc. The field data of the relationship type meets the preset field type. After selecting the part of the field data capable of constructing the relationship, the selected field data can be further scanned and analyzed to further filter the field data capable of constructing the relationship as target field data. Specifically, the field data can be filtered according to the specific content of the field data. For example, when the field data is a preset matching field, the field data meeting the preset matching field is determined as the target field data. Specifically, the preset matching field can be a serial number or an ID number, etc.

[0093] Step S13 includes vectorizing the field data of the multiple relationship types in the target field data to obtain the field vectors corresponding to the multiple field data respectively.

[0094] When the field data is vectorized, only the selected target field data is vectorized, thereby reducing the data processing amount and improving the data processing efficiency.

[0095] On the basis of the above technical solutions, step S15 can specifically include:

[0096] According to the distance between the field vectors corresponding to any two field data in the multiple field data, an association relationship between the multiple field data is established to obtain a field knowledge graph. The field knowledge graph includes multiple association relationship groups, and each association relationship group is composed of multiple field data establishing the association relationship.

[0097] According to the field knowledge graph and the multiple field data contained in the table data of each table file, an association relationship between the multiple table files is established to obtain a table file knowledge graph.

[0098] In the embodiment, according to the distance between the field vectors corresponding to any two field data in the plurality of field data, if the distance between the field vectors corresponding to any two field data is greater than a preset distance threshold, the data level recognition scanning of the field data can be performed, wherein the preset distance threshold can be 0.5, and the data level recognition scanning can be the value range boundary scanning and the coincidence degree scanning, thereby establishing the association relationship for the field data capable of establishing the association relationship, obtaining the field knowledge graph, and obtaining the field knowledge graph, which includes a plurality of association relationship groups, each association relationship group is composed of a plurality of field data capable of establishing the association relationship, so as to search the field data of the entire association relationship group according to one field data, facilitate data inventory, and on the basis of the field knowledge graph, according to the plurality of field data contained in the table data of each table file, the relationship between each field data and the table file is obtained, thereby further establishing the association relationship between a plurality of table files, obtaining a table file knowledge graph, and at this time, the table file knowledge graph contains the field knowledge graph.

[0099] On the basis of the above technical solution, the target field data further includes field data of a time sequence type.

[0100] After the association relationship between a plurality of table files is established according to the field knowledge graph and the plurality of field data contained in the table data of each table file, to obtain a table file knowledge graph, the method further includes:

[0101] According to the field data of the relationship type and the field data of the time sequence type in each table file, the association relationship between the field data in each table file is described in time sequence, and a table file knowledge graph with time sequence is obtained.

[0102] On the basis of the above technical solution, after the association relationship between the field data in each table file is described in time sequence according to the field data of the relationship type and the field data of the time sequence type in each table file, to obtain a table file knowledge graph with time sequence, the method further includes:

[0103] According to the table file knowledge graph with time sequence, a system interface for data inventory management is generated.

[0104] According to the constraint condition of the system interface and the table file knowledge graph with time sequence, a business theme is generated.

[0105] The constraint condition is a corresponding constraint condition input by a user according to an actual business situation, which is not specifically limited here.

[0106] Through the method, the association relationship at the data level can be converted into a visual association relationship, so as to perform data inventory.

[0107] Figure 2is a flow chart of a method of vectorizing field data according to an exemplary embodiment, as Figure 2 As shown in the above technical solution, on the basis of the above technical solution, the specific method of vectorizing the plurality of field data to obtain the field vector corresponding to each of the plurality of field data can include the following steps:

[0108] In step S21, according to the plurality of table files, the number of elements expected to be supported by the filter and the filter bit array value are determined.

[0109] In this embodiment, according to the plurality of table files, the maximum row value in the plurality of table files is determined as the number of elements expected to be supported by the filter, and then the filter bit array value is determined according to the number of elements expected to be supported by the filter. The specific calculation method can be: filter bit array value = (1 + n / 100 million rows) x 4MB, wherein n is the number of elements expected to be supported by the filter.

[0110] In step S22, the number K of hash functions is determined according to the number of elements expected to be supported by the filter, the false positive probability and the filter bit array value.

[0111] The calculation formula for calculating the number K of hash functions can be:

[0112]

[0113] Wherein, fpp is the false positive probability, the value of the false positive probability can be one ten-thousandth, n is the number of elements expected to be supported by the filter, and m is the filter bit array value.

[0114] Through the above formula, the value of the number K of hash functions can be calculated according to the number of elements expected to be supported by the filter, the false positive probability and the filter bit array value.

[0115] In step S23, for each field data, the field data is mapped into K points in a bit array by K hash functions, and 0 in the K points in the bit array is replaced by 1. The positions of the K points obtained by different field data are not completely the same.

[0116] The specific method of determining the position of the K points corresponding to each field data can use Murmur hash (non-encryption hash function) to determine.

[0117] In step S24, according to the bit array mapped by each field data, the field vector corresponding to each field data is obtained.

[0118] Through the above method, a binary bit array can be obtained, and the field vector corresponding to each field data can be obtained through the binary bit array mapped by each field data.

[0119] On the basis of the above technical solutions, the distance between the field vectors corresponding to any two field data is obtained according to the field vectors corresponding to the respective field data, and the distance calculation method includes at least one of the following: Euclidean distance, Manhattan distance, Chebyshev distance, Hamming distance and cosine distance.

[0120] The distance between the field vectors corresponding to any two field data is calculated in the field vectors corresponding to the respective field data, wherein the distance calculation method includes at least one of the following: Euclidean distance, Manhattan distance, Chebyshev distance, Hamming distance and cosine distance.

[0121] The distance between the field vectors corresponding to any two field data is calculated according to the Hamming distance, which can specifically include the following steps:

[0122] The number of 1s in the same point position of the field vectors corresponding to any two field data is obtained.

[0123] For the field vectors corresponding to any two field data, the data in the same point position is judged, and if it is 1, it is counted once, so that the number of 1s in the same point position of the field vectors corresponding to any two field data is counted.

[0124] The similarity value of the two field data is determined according to the number of 1s in the same point position of the field vectors corresponding to the two field data, wherein the number of 1s in the same point position of the field vectors corresponding to the two field data is proportional to the similarity value.

[0125] In this embodiment, the greater the number of 1s in the same point position of the field vectors corresponding to the two field data, the greater the similarity value, and the value range of the similarity value is 0 to 1.

[0126] The similarity value of the two field data is determined as the distance between the field vectors corresponding to the two field data.

[0127] The similarity value of the two field data is determined as the distance between the field vectors corresponding to the two field data, and at this time, the value range of the distance between the field vectors corresponding to the two field data is also 0 to 1.

[0128] Figure 3 is a flow chart of a method for constructing data modeling association according to an exemplary embodiment, as shown in Figure 3 The field data vector is the field vector corresponding to the field data, the table file association relationship is the association relationship between the plurality of table files, the table file business theme is the generated business theme, S1, S2 and S3 are the order of the steps, S1 is executed first, then S2 is executed, and finally S3 is executed.

[0129] Figure 4is a block diagram of an apparatus for constructing data modeling association according to an exemplary embodiment. Referring to Figure 4 The apparatus comprises a scanning module 41, an obtaining module 42, a first obtaining module 43, a second obtaining module 44 and a establishing module 45.

[0130] The scanning module 41 is configured to scan a plurality of table files in a plurality of databases contained in a target system to obtain a plurality of library table data respectively corresponding to the plurality of table files, wherein the library table data at least comprises business data.

[0131] The obtaining module 42 is configured to obtain table data of each table file and a plurality of field data contained in the table data according to the business data of each table file.

[0132] The first obtaining module 43 is configured to vectorize the plurality of field data to obtain a plurality of field vectors respectively corresponding to the plurality of field data.

[0133] The second obtaining module 44 is configured to obtain a distance between field vectors respectively corresponding to any two field data according to the plurality of field vectors respectively corresponding to the plurality of field data.

[0134] The establishing module 45 is configured to establish an association relationship between the plurality of table files according to the distance between the field vectors respectively corresponding to any two field data in the plurality of field data and the plurality of field data contained in the table data of each table file.

[0135] Optionally, the library table data further comprises table creation description, and the table creation description of each table file comprises a table structure of the table file and a field type of each field data contained in the table file.

[0136] Before the first obtaining module, the apparatus further comprises:

[0137] A third obtaining module is configured to obtain target field data from the plurality of field data according to the field type of each field data contained in each table file, wherein the target field data at least comprises relationship type field data.

[0138] The first obtaining module comprises:

[0139] A first obtaining submodule is configured to vectorize the plurality of relationship type field data in the target field data to obtain a plurality of field vectors respectively corresponding to the plurality of field data.

[0140] Optionally, the establishing module comprises:

[0141] The first establishing submodule is configured to establish an association relationship among the plurality of field data according to a distance between field vectors corresponding to any two field data in the plurality of field data, so as to obtain a field knowledge graph, wherein the field knowledge graph comprises a plurality of association relationship groups, and each association relationship group is composed of a plurality of field data with established association relationship.

[0142] The second establishing submodule is configured to establish an association relationship among the plurality of table files according to the field knowledge graph and a plurality of field data contained in table data of each table file, so as to obtain a table file knowledge graph.

[0143] Optionally, the target field data further comprises field data of a time sequence type.

[0144] After the second establishing submodule, the device further comprises:

[0145] The fourth obtaining module is configured to perform time sequence description on the association relationship among the field data in each table file according to field data of a relationship type and field data of a time sequence type in each table file, so as to obtain a table file knowledge graph with time sequence.

[0146] Optionally, after the fourth obtaining module, the device further comprises:

[0147] The first generating module is configured to generate a system interface for data inventory management according to the table file knowledge graph with time sequence.

[0148] The second generating module is configured to generate a business theme according to a constraint condition of the system interface and the table file knowledge graph with time sequence.

[0149] Optionally, the first obtaining module comprises:

[0150] The first determining submodule is configured to determine a number of elements expected to be supported by the filter and a filter bit array value according to the plurality of table files.

[0151] The second determining submodule is configured to determine a number K of hash functions according to the number of elements expected to be supported by the filter, a false positive probability and the filter bit array value.

[0152] The mapping submodule is configured to, for each field data, map the field data into K point bits in a bit array through K hash functions, replace 0 in the K point bits in the bit array with 1, and the positions of the K point bits obtained by different field data are not completely same.

[0153] The obtaining submodule is configured to obtain a field vector corresponding to each field data according to a bit array obtained by each field data.

[0154] Optionally, the second obtaining module comprises:

[0155] a calculating sub-module, configured to calculate a distance between any two field vectors corresponding to any two field data in the field vectors corresponding to the plurality of field data, wherein the distance calculation method comprises at least one of the following: Euclidean distance, Manhattan distance, Chebyshev distance, Hamming distance and cosine distance.

[0156] With regard to the apparatus in the above-described embodiments, the specific manner in which the various modules perform operations has been described in detail in the embodiments of the method, and will not be described in detail here.

[0157] Figure 5 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 500 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0158] Referring to Figure 5 , the electronic device 500 can include one or more of the following components: a processing component 502, a memory 504, a power supply component 506, a multimedia component 508, an audio component 510, an input / output (I / O) interface 512, a sensor component 514, and a communication component 516.

[0159] The processing component 502 usually controls overall operations of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 can include one or more processors 520 to execute instructions to complete all or part of steps of the above-described methods. In addition, the processing component 502 can include one or more modules to facilitate interaction between the processing component 502 and other components. For example, the processing component 502 can include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0160] The memory 504 is configured to store various types of data to support operations of the electronic device 500. Examples of these data include instructions for any application or method operating on the electronic device 500, contact data, phonebook data, messages, pictures, videos, and the like. The memory 504 can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0161] The power component 506 provides power to the various components of the electronic device 500. The power component 506 can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the electronic device 500.

[0162] The multimedia component 508 includes a screen providing an output interface between the electronic device 500 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes the touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and pressure related to the touching or sliding action. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0163] The audio component 510 is configured to output and / or input an audio signal. For example, the audio component 510 includes a microphone (MIC) configured to receive an external audio signal when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting an audio signal.

[0164] The input / output interface 512 provides an interface between the processing component 502 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0165] The sensor component 514 includes one or more sensors for providing status assessments for various aspects of the electronic device 500. For example, the sensor component 514 can detect an open / closed position of the electronic device 500, relative positioning of components, such as a display and a keypad of the electronic device 500, a change in position of the electronic device 500 or a component of the electronic device 500, presence or absence of user contact with the electronic device 500, orientation or acceleration / deceleration of the electronic device 500, and temperature changes of the electronic device 500. The sensor component 514 can include an optical sensor for detecting ambient light, a proximity sensor configured to detect the presence of nearby objects without any physical touch, a CMOS or CCD image sensor for use in imaging applications, or an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor in some embodiments.

[0166] The communication component 516 is configured to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a cellular network (e.g., 2G, 3G, 5G, or 5G), or a combination thereof. In an example embodiment, the communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 516 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques, and other techniques.

[0167] In an example embodiment, the electronic device 500 can be implemented with one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements to perform the above-described method of constructing data modeling association.

[0168] In an example embodiment, a storage medium including instructions, such as the memory 504 including instructions, is also provided. The instructions can be executable by the processor 520 of the electronic device 500 to implement the above-described method of constructing data modeling association. Alternatively, the storage medium can be a non-transitory computer-readable storage medium, such as ROM, random access memory (RAM), CD-ROM, magnetic tapes, floppy disks, and optical data storage devices, and the like.

[0169] In an example embodiment, a computer program product is also provided, which includes readable program codes executable by the processor 520 of the electronic device 500 to accomplish the method of building data modeling association described above. Optionally, the program codes can be stored in a storage medium of the electronic device 500, which can be a non-transitory computer readable storage medium, for example, the non-transitory computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0170] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the present disclosure cover any and all variations of the present disclosure that come within the scope of the following claims and their equivalents. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0171] It is to be understood that the application is not limited to the precise details of construction and the arrangement of components described above and illustrated in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the application should only be limited by the claims appended hereto.

Claims

1. A method for constructing data modeling associations, characterized in that, The method includes: Scan multiple table files in multiple databases contained in the target system to obtain the database table data corresponding to each of the multiple table files, wherein the database table data includes at least business data; Based on the business data of each table file, obtain the table data of each table file and the multiple field data contained in the table data; Vectorize multiple fields of data to obtain field vectors corresponding to each field. Based on the field vectors corresponding to each of the multiple field data, the distance between the field vectors corresponding to any two field data can be obtained. Based on the distance between the field vectors corresponding to any two fields in the data and the multiple fields contained in the table data of each table file, establish the relationship between multiple table files; The process of vectorizing multiple field data to obtain field vectors corresponding to each field data includes: Based on multiple table files, determine the expected number of elements that the filter will support and the filter bit array value; The number of hash functions K is determined based on the expected number of elements supported by the filter, the false positive probability, and the filter bit array value. For each field of data, the field data is mapped to K points in a bit array using K hash functions, and the 0s in the K points in the bit array are replaced with 1s. The positions of the K points obtained for different field data are not exactly the same. Based on the bit array mapped to each field data, the field vector corresponding to each field data is obtained.

2. The method for constructing data modeling associations according to claim 1, characterized in that, The database table data also includes table creation descriptions. The table creation description of each table file includes the table structure of the table file and the field type of each field data contained in the table file. Before vectorizing multiple field data to obtain the field vectors corresponding to each field data, the method further includes: Based on the field type of each field data contained in each table file, the target field data is obtained from multiple field data, wherein the target field data includes at least field data of relation type; Vectorize multiple fields of data to obtain field vectors corresponding to each field, including: Vectorize the field data of multiple relation types in the target field data to obtain the field vectors corresponding to each of the multiple field data.

3. The method for constructing data modeling associations according to claim 2, characterized in that, Based on the distance between the field vectors corresponding to any two fields in multiple data sets and the multiple fields contained in the table data of each table file, establish the relationships between multiple table files, including: Based on the distance between the field vectors corresponding to any two field data in multiple field data, an association relationship is established between multiple field data to obtain a field knowledge graph. The field knowledge graph includes multiple association relationship groups, and each association relationship group consists of multiple field data that have established association relationships. Based on the field knowledge graph and the multiple field data contained in the table data of each table file, establish the association relationship between multiple table files to obtain the table file knowledge graph.

4. The method for constructing data modeling associations according to claim 3, characterized in that, The target field data also includes time-series type field data; After establishing relationships between multiple table files based on the field knowledge graph and the multiple field data contained in the table data of each table file to obtain the table file knowledge graph, the method further includes: Based on the relational and temporal field data in each table file, the relationships between the field data in each table file are described in a temporal sequence, resulting in a table file knowledge graph with temporal sequence characteristics.

5. The method for constructing data modeling associations according to claim 4, characterized in that, After describing the relationships between fields in each table file in a time-series manner based on the relational and temporal type field data, thus obtaining a time-series table file knowledge graph, the method further includes: Generate a system interface for data inventory management based on a knowledge graph of time-series table files; Based on the constraints of the system interface and the time-series table file knowledge graph, business topics are generated.

6. The method for constructing data modeling associations according to claim 1, characterized in that, Based on the field vectors corresponding to each of the multiple field data points, the distance between the field vectors corresponding to any two field data points is obtained, including: Calculate the distance between any two field vectors corresponding to multiple field data. The distance calculation method used includes at least one of the following: Euclidean distance, Manhattan distance, Chebyshev distance, Hamming distance, and cosine distance.

7. An apparatus for constructing data modeling associations, characterized in that, The device includes: The scanning module is used to scan multiple table files in multiple databases contained in the target system to obtain the database table data corresponding to each of the multiple table files, wherein the database table data includes at least business data. The acquisition module is used to acquire the table data of each table file and the multiple field data contained in the table data, based on the business data of each table file. The first acquisition module is used to vectorize multiple field data to obtain the field vectors corresponding to each of the multiple field data. The second obtaining module is used to obtain the distance between the field vectors corresponding to any two field data based on the field vectors corresponding to each of the multiple field data. The module is used to establish relationships between multiple table files based on the distance between the field vectors corresponding to any two field data in multiple field data and the multiple field data contained in the table data of each table file; The first obtaining module includes: The first determining submodule is used to determine the expected number of elements supported by the filter and the filter bit array value based on multiple table files; The second determining submodule is used to determine the number K of hash functions based on the expected number of elements supported by the filter, the false positive probability, and the filter bit array value; The mapping submodule is used to map each field of data to K bits in a bit array using K hash functions, and replace 0s with 1s in the K bits of the bit array. The positions of the K bits obtained for different field data are not exactly the same. The `obtain` submodule is used to obtain the field vector corresponding to each field data based on the bit array mapped to each field data.

8. An electronic device, characterized in that, include: processor; Memory used to store instructions executed by the processor; The processor is configured to execute the instructions to implement the method for constructing data modeling associations as described in any one of claims 1 to 6.

9. A storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the method of constructing data modeling associations as described in any one of claims 1 to 6.

Images