Data processing method and device, electronic equipment and storage medium

Abstract

The application provides a data processing method and device and electronic equipment, and relates to the technical field of data processing. The method comprises the following steps: acquiring a plurality of file name sets; acquiring a plurality of data sets; for any file name set, determining a target data set similar to the any file name set from the plurality of data sets according to each file name in the any file name set; and for any data field in the target data set, if there is a file name in the any file name set that matches the any data field, generating association blood relationship information according to the table name of the data table where the target data set is located, the any data field and the file name. In this way, the similarity of the file name of the clustered unstructured file in the data lake and the data field in the same column or the same row in the structured data table can be calculated to obtain the association blood relationship information, so that the mapping association information between the structured data table and the unstructured file can be effectively established.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, electronic device and storage medium. Background Technology

[0002] A data lake is a repository or system that stores data in its raw format, storing data as is without prior structuring. In data lake operations, there are situations where structured data (such as data tables) and unstructured data (such as files) need to be associated. For example, if personal information is recorded in a data table, it is necessary to associate this personal information with unstructured data (such as photos of an individual's ID card, vehicle photos, other text files, etc.), requiring the unstructured data to be stored in object storage.

[0003] Establishing the lineage information (i.e., mapping relationship information) between structured data (such as data tables) and unstructured data (such as files) is very important. Summary of the Invention

[0004] The purpose of this application is to at least partially solve one of the aforementioned technical problems.

[0005] To this end, this application proposes a data processing method, apparatus, electronic device, and storage medium to perform similarity calculations on the filenames of unstructured files clustered in a data lake and data fields in the same column or row of a structured data table to obtain related lineage information. This can effectively establish mapping and association information between structured data tables and unstructured files, improving the effectiveness of querying or accessing unstructured files.

[0006] The first aspect of this application provides a data processing method, including:

[0007] Obtain multiple sets of first filenames, wherein the first set of filenames includes the filenames of each file in the same cluster, and the cluster is obtained by clustering the files in at least one bucket in the data lake;

[0008] Obtain multiple first datasets, wherein the first datasets include data fields in the same column or row of the same data table in the data lake;

[0009] For any first file name set, based on each file name in the first file name set, determine a target dataset similar to the first file name set from the plurality of first datasets;

[0010] For any first data field in the target dataset, if there is a first filename in the set of first filenames that matches any first data field, first related lineage information is generated based on the table name of the data table where the target dataset is located, the first data field, and the first filename.

[0011] A second aspect of this application provides a data processing apparatus, comprising:

[0012] The first acquisition module is used to acquire multiple first file name sets, wherein the first file name set includes the file names of each file in the same cluster, and the cluster is obtained by clustering each file in at least one storage bucket in the data lake;

[0013] The second acquisition module is used to acquire multiple first datasets, wherein the first datasets include data fields in the same column or row of the same data table in the data lake;

[0014] A determination module is used to determine, for any first file name set, a target dataset similar to the first file name set from the plurality of first datasets based on each file name in the first file name set;

[0015] The generation module is used to generate first related lineage information based on the table name of the data table where the target dataset is located, the first data field, and the first file name, provided that a first file name in the set of first file names matches the first data field.

[0016] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data processing method as described in the first aspect.

[0017] The fourth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the data processing method as described in the first aspect.

[0018] A fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method described in the first aspect of this application.

[0019] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0020] By acquiring multiple sets of first filenames and multiple first datasets, and for any given set of first filenames, a target dataset similar to any given set of first filenames is determined from the multiple first datasets. For any first data field in the target dataset, if a first filename in any set of first filenames matches any first data field, first association lineage information is generated based on the table name of the data table containing the target dataset, the first data field, and the first filename. This allows for similarity calculation of filenames of unstructured files clustered in a data lake and data fields in the same column or row of structured data tables to obtain association lineage information. This effectively establishes a mapping relationship between structured data tables and unstructured files, improving the effectiveness of querying or accessing unstructured files.

[0021] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0022] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0023] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0024] Figure 2 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0025] Figure 3 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0026] Figure 4 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0027] Figure 5 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the structure of a data processing apparatus according to an embodiment of this application;

[0029] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0030] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0031] By recording filenames stored in object storage in a data table, the file corresponding to that filename in object storage can be accessed and retrieved by first looking up the filename in the data table and then using that filename.

[0032] Because object storage cannot be manipulated using SQL (Structured Query Language) statements, and filename records in data tables are implemented through applications and are not perceptible at the data layer, existing data lineage implementation methods cannot effectively establish mapping and association information (i.e., association lineage information) between data tables and unstructured data (such as files).

[0033] To address the aforementioned problems, this application proposes a data processing method, apparatus, and electronic device. Before describing the embodiments of this application in detail, for ease of understanding, commonly used technical terms are first introduced:

[0034] Object storage, also known as object-based storage, is a general term used to describe methods for solving and processing discrete units, which are called objects. Objects are similar to files in that they both contain data. However, unlike files, objects do not have a hierarchical structure within a single layer. Each object resides at the same level in a flat address space called a storage pool; one object does not belong to a lower level of another object.

[0035] Both files and objects have metadata related to the data they contain, with objects characterized by extended metadata. Each object is assigned a unique identifier that allows a server or end user to retrieve the object without needing to know the physical address of the data.

[0036] In object storage, a folder is merely a logical concept. When setting up a folder using an API (Application Programming Interface) or SDK (Software Development Kit), you can specify the key value corresponding to the object (e.g., abc / 1.jpg) to logically create a folder. For example, defining the key of the object as abc / 1.jpg will create a folder named "abc" under the bucket, and within that folder, there will be a file named "1.jpg".

[0037] In object storage, a folder is actually an empty file with a size of 0KB. Therefore, when a user creates an object with the key value 1 / , the folder 1 will be defined. Furthermore, if a user creates a file abc / 1.jpg, the system will not create the folder abc / . Therefore, after deleting abc / 1.jpg, the folder abc will no longer exist.

[0038] It's important to note that because object storage uses a distributed storage approach, objects are not physically stored according to folders; that is, not all files in a folder are stored together. During backend storage, files in different folders differ only in their key prefixes. Therefore, this architecture makes it difficult to easily summarize information about a folder, such as folder size or access frequency. To iterate through all files in a folder, you first need to obtain the keys of all files in that folder using the ListObject interface (specifying the folder by the prefix) before performing operations. Accessing files in object storage is done through REST interfaces, using HTTP (Hypertext Transfer Protocol) verbs (GET, POST, PUT, DELETE, etc.) to describe the operations.

[0039] The following is combined Figure 1 This application provides a detailed description of the data processing methods provided.

[0040] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application.

[0041] The data processing method of this application embodiment can be executed by the data processing apparatus provided in this application embodiment. The data processing apparatus in this application can be applied to an electronic device to perform data processing functions. Alternatively, the data processing apparatus can be configured in an application of an electronic device so that the application can perform data processing functions.

[0042] The electronic device can be any device with computing capabilities, and the device or the application within the device can perform data processing functions. Devices with computing capabilities can be, for example, personal computers (PCs), mobile terminals, servers, etc. Mobile terminals can be, for example, in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, and other hardware devices with various operating systems, touchscreens, and / or displays.

[0043] like Figure 1 As shown, the data processing method includes the following steps:

[0044] Step S101: Obtain multiple sets of first filenames, wherein the multiple sets of first filenames include the filenames of each file in the same cluster, and the cluster is obtained by clustering the files in at least one bucket in the data lake.

[0045] In this application embodiment, the files may include, but are not limited to: image files, document files, PDF (Portable Document Format) files, audio files, video files, etc.

[0046] In this embodiment, files in at least one bucket of a data lake can be clustered to obtain multiple clusters. As an example, for each bucket in the data lake, the files in that bucket can be clustered individually to obtain at least one cluster. Therefore, in this application, a first set of filenames can be generated based on the filenames of the files within the same cluster.

[0047] Step S102: Obtain multiple first datasets, wherein the first datasets include data fields in the same column or row of the same data table in the data lake.

[0048] In this embodiment of the application, if the attribute values ​​corresponding to each attribute field (name, age, data type, file name, etc.) are stored row by row in the data table, then the data fields in at least one data table in the data lake can be divided by row to obtain multiple first datasets, wherein each first dataset contains each data field in the same row of a data table.

[0049] In this embodiment of the application, if the attribute values ​​corresponding to each attribute field are stored in the data table by column, the data fields in at least one data table in the data lake can be divided by column to obtain multiple first datasets, wherein each first dataset contains each data field in the same column of a data table.

[0050] Step S103: For any first file name set, based on each file name in the first file name set, determine a target dataset similar to the first file name set from multiple first datasets.

[0051] In the embodiments of this application, for any first file name set in a plurality of first file name sets, a first dataset similar to the first file name set can be determined from a plurality of first datasets based on each file name in the first file name set, and used as the target dataset.

[0052] Step S104: For any first data field in the target dataset, if there is a first filename in any of the first filename sets that matches the first data field, generate first related lineage information based on the table name of the data table where the target dataset is located, the first data field, and the first filename.

[0053] In this embodiment of the application, for any data field in the target dataset (referred to as the first data field in this application), it can be determined whether there is a file name in any set of first file names that matches or is consistent with any first data field (i.e., the file name matches the first data field exactly). If there is a file name (referred to as the first file name in this application) in any set of first file names that matches or is consistent with any first data field, then the associated lineage information (i.e., mapping association information) can be generated based on the first file name, the table name of the data table where the target dataset is located, and the first data field.

[0054] As an example, for a certain first data field in the target dataset, the first data field can be used as a keyword to search any of the aforementioned first file name sets to determine whether there is a first file name in any of the first file name sets that matches or is consistent with the first data field. If there is a first file name in any of the first file name sets that matches or is consistent with the first data field, then the table name of the data table where the target dataset is located, the first data field, and the first file name that matches the first data field are recorded, and the association lineage information is formed.

[0055] As one possible approach, the related bloodline information can also be stored in a bloodline information database.

[0056] As another possible implementation, the related bloodline information can also be updated in the file metadata of the first file; that is, the related bloodline information can be added to the file metadata of the first file.

[0057] Therefore, it is possible to form a file association graph by associating lineage information, and then find related files based on the association graph. That is, the file metadata records the data table associated with the file, and by looking up the records in the data table, you can find similar files associated with that data table.

[0058] The data processing method of this application embodiment obtains multiple sets of first filenames and multiple sets of first datasets. For any set of first filenames, based on the filenames in the set, a target dataset similar to the set of first filenames is determined from the multiple first datasets. For any first data field in the target dataset, if a first filename in any set of first filenames matches any first data field, first association lineage information is generated based on the table name of the data table containing the target dataset, the first data field, and the first filename. Therefore, it is possible to obtain association lineage information by performing similarity calculations on the filenames of unstructured files clustered in a data lake and data fields in the same column or row of a structured data table. This effectively establishes a mapping association between structured data tables and unstructured files, improving the effectiveness of querying or accessing unstructured files.

[0059] To clearly illustrate how the target dataset similar to the first filename dataset is determined from multiple first datasets in the above embodiments of this application, this application also proposes a data processing method.

[0060] Figure 2 This is a flowchart illustrating another data processing method provided in an embodiment of this application.

[0061] like Figure 2 As shown, the data processing method may include the following steps:

[0062] Step S201: Obtain multiple sets of first filenames, wherein the first filename sets include the filenames of each file in the same cluster, and the cluster is obtained by clustering the files in at least one bucket in the data lake.

[0063] Step S202: Obtain multiple first datasets, wherein the first datasets include data fields in the same column or row of the same data table in the data lake.

[0064] The explanation of steps S201 to S202 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0065] Step S203: For any first file name set, determine the first similarity between the first file name set and each first dataset based on each file name in the first file name set.

[0066] In the embodiments of this application, for any first file name set in a plurality of first file name sets, a similarity calculation algorithm can be used to calculate the similarity between the first file name set and each first dataset based on each file name in the first file name set, which is denoted as the first similarity in this application.

[0067] In one possible implementation of this application, in order to improve the accuracy of the similarity calculation results, feature vectors of any first file name set and feature vectors of each first dataset can be obtained based on deep learning technology, so that the first similarity between any first file name set and each first dataset can be determined based on the similarity between the feature vectors.

[0068] As an example, features can be extracted from each filename in any first filename set to obtain the first text features of the first filename set. Features can also be extracted from each data field in each first dataset to obtain the second text features of each first dataset. The similarity between the first file features and each second text feature is calculated, which is referred to as the second similarity in this application. Thus, the first similarity between any first filename set and each first dataset can be determined based on the second similarity of each second text feature.

[0069] For example, for any second text feature, the second similarity of the second text feature can be used as the first similarity between the first dataset corresponding to the second text feature and the first file name set.

[0070] It should be noted that the number of data fields in the same row or column of the data table is large, that is, the number of data fields in the first dataset is large. If all data fields in the first dataset are included in the similarity calculation, it will consume a lot of computing resources and the computing efficiency will be low. Therefore, in view of the above situation, in one possible implementation of the embodiment of this application, the data fields in the first dataset can be sampled or sampled, and the similarity between the first file name set and the first dataset can be calculated only based on the sampled or sampled first dataset.

[0071] As an example, for any first dataset, each data field in the first dataset can be sampled to obtain the sampled first dataset. The sampled first dataset contains a set number of data fields (10,000, 5,000, etc.). Based on the data fields contained in the sampled first dataset and the file names in the first file name set, the first similarity between the first dataset and the first file name set can be calculated.

[0072] For example, features can be extracted from each data field contained in any of the sampled first datasets to obtain the second text features of any of the first datasets. Thus, in this application, the first similarity between any of the first filename sets and each of the first datasets can be determined based on the second similarity between the first text features and each of the second text features.

[0073] Therefore, sampling each first dataset and calculating similarity based on the sampled first dataset can not only improve computational efficiency but also save computational resources.

[0074] Step S204: Based on the first similarity of each first dataset, determine candidate datasets from each first dataset, wherein the first similarity of the candidate datasets is higher than a set similarity threshold.

[0075] In this embodiment of the application, for any first dataset, it can be determined whether the first similarity of the first dataset is greater than a set similarity threshold. If the first similarity of the first dataset is greater than the set similarity threshold, the first dataset can be used as a candidate dataset. If the first similarity of the first dataset is less than or equal to the similarity threshold, no processing is required.

[0076] Step S205: Determine the target dataset from the candidate datasets.

[0077] In this embodiment of the application, a target dataset similar to any of the first filename datasets described above can be determined from each candidate dataset.

[0078] In one possible implementation of this application, the target dataset can be determined, for example, by determining the number of hits of each filename in any first filename set within any candidate dataset. Thus, in this application, a target dataset similar to the first filename set can be determined from each candidate dataset based on the number of hits. For instance, the candidate dataset with the highest number of hits can be used as the target dataset similar to the first filename set.

[0079] As an example, for any candidate dataset, the above-mentioned first file name set and the candidate dataset can be stored in an analytical database (such as ES (Elasticsearch, a non-relational distributed full-text search framework)). Using the file names in the first file name as keywords, the candidate dataset is indexed in reverse to determine the number of times each file name in the first file name is matched in the candidate dataset. The candidate dataset with the highest number of matches is taken as the target dataset.

[0080] For example, if any of the first filename sets mentioned above includes 50 filenames, and candidate dataset 1 has 30 data fields that match the 30 filenames in any of the first filename sets, and candidate dataset 2 has 20 data fields that match the 20 filenames in any of the first filename sets, then candidate dataset 1 can be used as the target dataset.

[0081] In another possible implementation of the embodiments of this application, the target dataset may be determined as follows: the candidate dataset with the highest first similarity is taken as the target dataset that is similar to any of the first filename datasets mentioned above.

[0082] Therefore, it is possible to determine target datasets similar to any of the first filename sets mentioned above based on different methods, which can improve the flexibility and applicability of the method.

[0083] Step S206: For any first data field in the target dataset, if there is a first filename in any set of first filenames that matches any first data field, generate first related lineage information based on the table name of the data table where the target dataset is located, the first data field, and the first filename.

[0084] The explanation of step S206 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0085] The data processing method of this application embodiment determines a target dataset similar to the first file name set from each first dataset based on the similarity between the first file name set and each first dataset, which can improve the accuracy of the target dataset determination result.

[0086] To clearly illustrate how multiple sets of first filenames are obtained in any embodiment of this application, this application also proposes a data processing method.

[0087] Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application.

[0088] like Figure 3 As shown, the data processing method may include the following steps:

[0089] Step S301: For any bucket in the data lake, obtain the file information of each file in that bucket.

[0090] The file information includes at least one of the following: filename, file type, and file metadata. The file metadata may include, but is not limited to: creation time, creator, file size, number of hard links, access time, file metadata modification time, and file modification time.

[0091] In this embodiment of the application, for any bucket in the data lake, the file information of each file in that bucket can be obtained.

[0092] Step S302: Based on the file information of each file in any storage bucket, cluster the files in any storage bucket to obtain at least one cluster.

[0093] In this embodiment of the application, a clustering algorithm can be used to cluster the files in any of the aforementioned storage buckets based on the file information of each file, thereby obtaining at least one cluster.

[0094] Clustering algorithms include, but are not limited to: K-means clustering algorithm, KNN (K-Nearest Neighbor) classification algorithm, etc.

[0095] As an example, files in any given bucket can be clustered based on their filenames, so that files with similar filenames are grouped into the same cluster.

[0096] It should be noted that the file names in the data lake can be named according to the specified naming rules, or they can be named without following the specified naming rules. For example, they can be named according to the actual application requirements or business requirements. This application does not impose any restrictions on this.

[0097] As another example, files in any given bucket can be clustered according to their file types, so that files of the same type are clustered into the same cluster.

[0098] As another example, files in any bucket can be clustered based on their file metadata, so that files with similar file metadata are clustered into the same cluster.

[0099] Step S303: Generate a first filename set based on the filenames of each file in the same cluster.

[0100] In this embodiment of the application, the filenames of all files in the same cluster can be used as a first filename set.

[0101] Step S304: Obtain multiple first datasets, wherein the first datasets include data fields in the same column or row of the same data table in the data lake.

[0102] Step S305: For any first file name set, based on each file name in the first file name set, determine a target dataset similar to the first file name set from multiple first datasets.

[0103] Step S306: For any first data field in the target dataset, if there is a first filename in the set of first filenames that matches the first data field, generate first related lineage information based on the table name of the data table where the target dataset is located, the first data field, and the first filename.

[0104] The explanation of steps S304 to S306 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0105] The data processing method of this application embodiment can cluster the files in the storage bucket according to the file name, file type or file metadata of the files in the storage bucket to obtain multiple first file name sets. That is, by clustering the files in the storage bucket according to different file information, the flexibility and applicability of the method can be improved.

[0106] To clearly illustrate how multiple first datasets are obtained in any embodiment of this application, this application also proposes a data processing method.

[0107] Figure 4 This is a flowchart illustrating another data processing method provided in an embodiment of this application.

[0108] like Figure 4 As shown, the data processing method may include the following steps:

[0109] Step S401: Obtain multiple sets of first filenames, wherein the first filename sets include the filenames of each file in the same cluster, and the cluster is obtained by clustering the files in at least one bucket in the data lake.

[0110] The explanation of step S401 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0111] Step S402: Obtain multiple initial datasets, wherein each initial dataset includes all data fields in the same column or row of the same data table in the data lake.

[0112] In this embodiment of the application, if the attribute values ​​corresponding to each attribute field (name, age, data type, file name, etc.) are stored row by row in the data table, then the data fields in at least one data table in the data lake can be divided by row to obtain multiple initial datasets, wherein each initial dataset includes all data fields in the same row of the same data table.

[0113] In this embodiment of the application, if the attribute values ​​corresponding to each attribute field are stored in the data table by column, the data fields in at least one data table in the data lake can be divided by column to obtain multiple initial datasets, wherein each initial dataset contains all data fields in the same column of the same data table.

[0114] Step S403: For any initial dataset, determine the retention score of the initial dataset based on the repetition rate of each data field in the initial dataset.

[0115] Among them, the retention score is negatively correlated with the repetition rate. That is, the lower the repetition rate, the higher the retention score, and vice versa.

[0116] In this embodiment, for any initial dataset among multiple initial datasets, the repetition rate of each data field in that initial dataset can be determined. For example, if any initial dataset contains 50 data fields and 40 data fields are repeated, then the repetition rate of that initial dataset can be determined to be 0.8. In this application, the retention score of any initial dataset can be determined based on the repetition rate of each data field in that initial dataset. The retention score is negatively correlated with the repetition rate; that is, the lower the repetition rate, the higher the retention score, and vice versa.

[0117] Step S404: Determine each first dataset from each initial dataset based on the retained scores of each initial dataset.

[0118] In this embodiment of the application, each first dataset can be determined from each initial dataset based on the retention score of each initial dataset.

[0119] As an example, the initial dataset with scores higher than a set score threshold can be used as the first dataset.

[0120] As another example, the initial datasets can be sorted from largest to smallest according to the value of the retained score, and the first set number of initial datasets (such as 2, 3, 4, etc.) can be selected as the first dataset.

[0121] For example, data fields in a data table that are repetitive, such as enumeration values, boolean values, and date types, cannot be used as data fields to match filenames. Therefore, the initial dataset containing these data fields can be filtered to retain the initial dataset containing data fields that may be used as filenames.

[0122] Step S405: For any first file name set, based on each file name in the first file name set, determine a target dataset similar to the first file name set from multiple first datasets.

[0123] Step S406: For any first data field in the target dataset, if there is a first filename in the set of first filenames that matches the first data field, generate first related lineage information based on the table name of the data table where the target dataset is located, the first data field, and the first filename.

[0124] The explanation of steps S405 to S406 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0125] The data processing method in this application embodiment can filter each initial dataset to remove initial datasets containing data fields that cannot be used as filenames. On the one hand, this can reduce the amount of computation, and on the other hand, it can improve the accuracy of establishing related bloodline information.

[0126] To clearly illustrate any embodiment of this application, a data processing method is also proposed.

[0127] Figure 5 This is a flowchart illustrating another data processing method provided in an embodiment of this application.

[0128] like Figure 5 As shown, in Figures 1 to 4 Based on any of the embodiments shown, the data processing method may further include the following steps:

[0129] Step S501: Mark any of the first data fields mentioned above in the data table where the target dataset is located.

[0130] In this embodiment of the application, after comparing each first file name set with a target dataset similar to the first file name set, and associating the file name with the data field and table name, the file name and data field that are not associated or do not have a matching file name can be extracted. That is, the associated file name and data field are excluded, and the unassociated file name and data field are extracted.

[0131] As an example, the first data field in the data table containing the target dataset that has associated filenames can be marked. The purpose of marking is to clarify that the first data field has associated filenames.

[0132] Step S502: Mark the file corresponding to the first filename in at least one storage bucket.

[0133] Similarly, files in at least one bucket in the data lake that correspond to the first filename can be marked. The purpose of marking is to clarify that the file corresponding to the first filename has associated data fields and data tables.

[0134] Step S503: Based on the unlabeled data fields in each data table and the unlabeled files in each storage bucket, perform at least one round of iteration until the set stop iteration condition is met.

[0135] In this embodiment of the application, the stopping iteration condition may include at least one of the following:

[0136] The first condition is that there are no untagged files in any bucket, meaning that the filenames of all files in the bucket are associated with data fields in the data table, or the number of untagged files is less than the set first threshold.

[0137] The second condition is that there are no unlabeled data fields in each of the first datasets, that is, all data fields in the first dataset have associated filenames, or the number of unlabeled data fields is less than the set second quantity threshold.

[0138] Thirdly: The filenames of all unlabeled files do not match or are inconsistent with the unlabeled data fields in the first dataset.

[0139] Fourthly: In any iteration, before matching or calculating the similarity between each set of second filenames and each set of second datasets, ensure that the number of filenames contained in each set of second filenames is less than the set third quantity threshold.

[0140] In this embodiment of the application, the first iteration process may include the following steps:

[0141] 1. Cluster the unlabeled files in each storage bucket to obtain at least one cluster, and generate at least one second file name set corresponding to the first iteration process based on the file names of each file in the at least one cluster. That is, the file names of each file in the same cluster can be used as a second file name set.

[0142] 2. Divide the unlabeled data fields in each data table by column or row to obtain at least one second dataset corresponding to the first iteration process.

[0143] 3. For any set of second filenames, based on each filename in the set of second filenames, determine the second datasets that are similar to the set of second filenames from each set of second datasets, and use them as similar datasets.

[0144] 4. For any second data field in the similar dataset, if there is a second filename in any of the above-mentioned second filename sets that matches the second data field, generate second related lineage information based on the table name of the data table where the similar dataset is located, the second data field, and the second filename.

[0145] 5. Mark the second data field of the data table containing the similar datasets that has the associated file name, so as to obtain the unmarked data fields in each data table updated in this round of iteration.

[0146] 6. Mark the files in each bucket that correspond to the second filename of the associated second data field, so as to obtain the unmarked files in each bucket obtained in this iteration process.

[0147] For processes that are not the first iteration, the following steps may be included:

[0148] 1. Based on the unlabeled data fields in each data table and the unlabeled files in each storage bucket obtained from the previous iteration process, determine whether the set iteration conditions are met. If yes, stop the iteration; otherwise, proceed to the next step.

[0149] 2. Cluster the unlabeled files in each bucket obtained from the previous iteration process to obtain at least one cluster. Based on the file names of each file in the at least one cluster, generate at least one second file name set corresponding to the current iteration process. That is, the file names of each file in the same cluster can be used as a second file name set.

[0150] 3. Divide the unlabeled data fields in each data table obtained from the previous iteration process by column or row to obtain at least one second dataset corresponding to the current iteration process.

[0151] 4. For any set of second filenames, based on each filename in the set of second filenames, determine the second datasets that are similar to the set of second filenames from each set of second datasets, and use them as similar datasets.

[0152] 5. For any second data field in the similar dataset, if there is a second filename in any of the above second filename sets that matches the second data field, generate second related lineage information based on the table name of the data table where the similar dataset is located, the second data field, and the second filename.

[0153] 6. Mark the second data field of the data table containing the similar datasets that has the associated file name, so as to obtain the unmarked data fields in each data table updated in this round of iteration.

[0154] 7. Mark the files in each bucket that correspond to the second filename of the associated second data field, so as to obtain the unmarked files in each bucket obtained in this iteration process.

[0155] It should be noted that in practical applications, the stopping iteration condition can also be other conditions. For example, the stopping iteration condition can also include: any second file name set in any round of iteration is not similar to each second dataset. For example, the similarity between any second file name set and each second dataset is less than the set similarity threshold.

[0156] In summary, it is possible to perform iterative calculations on unlabeled file objects based on unlabeled data fields, thereby improving the richness and completeness of the generated related lineage information.

[0157] In any embodiment of this application, unstructured data stored in a data lake can be categorized based on storage buckets using intelligent algorithms. Similarity analysis is then performed between this classification and the data fields in the data tables to identify columns that might record filenames. Finally, based on keyword-based reverse indexing, the corresponding data tables and columns are accurately retrieved, thereby establishing the correlation between data tables and unstructured data. Specifically, this may include the following steps:

[0158] Step 1: For the unstructured data stored in each bucket in the data lake, based on the file name, file type, or other metadata information (including creation time, creator, etc.), use machine learning clustering algorithms, such as k-means clustering, to cluster and classify the unstructured data into multiple file sets (referred to as clusters in this application), and extract the file names of each file in the file set to obtain the file name set.

[0159] Step 2: Compare the filename sets with the data tables stored in the data lake. First, filter the data fields in the data tables to exclude data fields that cannot be used as filenames, such as enumerated values, boolean values, date types, and other repetitive data fields. Then, for each column that can participate in the calculation, create a dataset. At this point, there will be multiple datasets. Considering the huge amount of data and the impact on calculation speed, this application can perform sampling calculations when the amount of data fields in the dataset is too large, for example, extracting 10,000 rows of data fields as dataset samples. Afterward, similarity calculations can be performed on the filename sets and datasets, for example, by using the embedding algorithm in deep learning to calculate the similarity between each filename set and the dataset, to obtain the datasets similar to each filename set, and obtain the table name of the data table containing the datasets similar to the filename sets.

[0160] Step 3: When a set of filenames is similar to multiple datasets, these datasets can be further analyzed to find a more accurate dataset. For example, the set of filenames and datasets that need further comparison can be stored in an Elasticsearch (ES) analytical database. Using filenames as keywords, the datasets can be indexed using inverted indexes to obtain the matching rate of each filename in the set with similar datasets. The dataset with the highest matching rate or number of matching occurrences can be used as the final dataset to be associated with.

[0161] Step 4: After obtaining the filename set, the dataset of similar filenames, and the table name of the data table containing the dataset, use the data field in the data table as the keyword to search the filename set to determine if there is a filename in the filename set that matches or is identical to the data field. If a filename that matches the data field exists, record the data table, the data field, and the existing filename to form the related lineage information, and add it to the lineage information database. Furthermore, the related lineage information can be synchronously updated in the file metadata. If no filename that matches the data field exists, initially set it to null.

[0162] Step 5: After comparing and associating each filename set, associated data table, and data field, a lineage information is established for filenames, data tables, and data fields that are associated. For filenames and data fields that are not associated, the data is extracted, and the already associated filenames and data fields are excluded. By repeating steps 1 to 4, the remaining data is iteratively calculated, and further associations are attempted for the remaining data until the filename set and dataset have a large degree of similarity, at which point the iterative calculation stops.

[0163] Step 6: Through multiple iterations, most of the unstructured data can eventually establish a lineage with the corresponding data table.

[0164] Step 7: The above is a method for self-learning of association lineage without human intervention. If the file names of each file in the data lake are named according to the specified naming rules, this method can simplify steps 2 and 3 above and speed up the process of establishing association lineage. However, it requires high quality requirements for the data tables entering the data lake.

[0165] In summary, within a data lake, related lineage information can be obtained by analyzing unstructured data and data tables using intelligent algorithms.

[0166] Corresponding to the data processing methods provided in the above embodiments, one embodiment of this application also provides a data processing apparatus. Since the data processing apparatus provided in this embodiment corresponds to the data processing methods provided in the above embodiments, the implementation methods of the data processing methods are also applicable to the data processing apparatus provided in this embodiment, and will not be described in detail here.

[0167] Figure 6 This is a schematic diagram of the structure of a data processing apparatus according to an embodiment of the present application.

[0168] like Figure 6 As shown, the data processing device 600 may include: a first acquisition module 601, a second acquisition module 602, a determination module 603, and a generation module 604.

[0169] The first acquisition module 601 is used to acquire multiple first file name sets, wherein the first file name sets include the file names of each file in the same cluster, and the cluster is obtained by clustering the files in at least one storage bucket in the data lake.

[0170] The second acquisition module 602 is used to acquire multiple first datasets, wherein the first datasets include data fields in the same column or row of the same data table in the data lake.

[0171] The determination module 603 is used to determine, for any first file name set, a target dataset similar to any first file name set from multiple first datasets based on each file name in any first file name set.

[0172] The generation module 604 is used to generate first related lineage information based on the table name of the data table where the target dataset is located, any first data field, and the first file name, provided that a first file name in any set of first file names matches any first data field.

[0173] As one possible implementation of this application, the determining module 603 is specifically used for: for any first file name set, determining a first similarity between any first file name set and each first dataset based on each file name in the first file name set; determining candidate datasets from each first dataset based on the first similarity of each first dataset, wherein the first similarity of the candidate datasets is higher than a set similarity threshold; and determining a target dataset from each candidate dataset.

[0174] As one possible implementation of this application, the determining module 603 is specifically used for: determining the number of hits of each filename in any first filename set in any candidate dataset for any candidate dataset; determining the target dataset from each candidate dataset based on the number of hits in each candidate dataset; or, taking the candidate dataset with the highest first similarity as the target dataset.

[0175] As one possible implementation of this application, the determining module 603 is specifically used for: extracting features from each filename in any first filename set to obtain a first text feature of any first filename set; extracting features from each data field in each first dataset to obtain a second text feature of each first dataset; and determining a first similarity between any first filename set and each first dataset based on a second similarity between the first text feature and each second text feature.

[0176] As one possible implementation of this application, the determining module 603 is specifically used for: sampling each data field in any first dataset to obtain a sampled first dataset, wherein the sampled first dataset contains a set number of data fields; and extracting features from each data field contained in the sampled first dataset to obtain the second text features of the first dataset.

[0177] As one possible implementation of this application, the second acquisition module 602 is specifically used for: acquiring multiple initial datasets, wherein each initial dataset includes all data fields in the same column or row of the same data table in the data lake; for any initial dataset, determining the retention score of any initial dataset based on the repetition rate of each data field in the initial dataset, wherein the retention score is negatively correlated with the repetition rate; and determining each first dataset from each initial dataset based on the retention score of each initial dataset.

[0178] As one possible implementation of this application, the first acquisition module 601 is specifically used for: acquiring file information of each file in any bucket in the data lake; clustering each file in any bucket according to the file information of each file in the bucket to obtain at least one cluster; generating a first file name set according to the file names of each file in the same cluster; wherein the file information includes at least one of the following: file name, file type and file metadata.

[0179] As one possible implementation of this application embodiment, the data processing apparatus 600 may further include:

[0180] The update module is used to update the file metadata of the file corresponding to the first filename based on the first related lineage information.

[0181] As one possible implementation of this application embodiment, the data processing apparatus 600 may further include:

[0182] The first marking module is used to mark any first data field in the data table where the target dataset is located.

[0183] The second marking module is used to mark the file corresponding to the first filename in at least one storage bucket.

[0184] The execution module is used to perform at least one round of iteration based on the unlabeled data fields in each data table and the unlabeled files in each storage bucket, until the set stopping iteration condition is met.

[0185] The stopping iteration conditions include at least one of the following: there are no unlabeled files in each bucket; there are no unlabeled data fields in each first dataset; and the unlabeled data fields in the first dataset do not match the filenames of the unlabeled files.

[0186] As one possible implementation of this application embodiment, any round of iteration process includes:

[0187] Obtain at least one second file name set, wherein the second file name set includes the file names of each file in the same file set, and the file set is obtained by clustering the unlabeled files in each bucket obtained from the previous iteration process;

[0188] Obtain at least one second dataset, wherein the second dataset includes unlabeled data fields in the same column or row of the same data table obtained from the previous iteration process;

[0189] For any set of second filenames, based on each filename in the set of second filenames, determine similar datasets from each set of second datasets that are similar to the set of second filenames.

[0190] For any second data field in a similar dataset, if there is a second filename in any second filename set that matches any second data field, generate second related lineage information based on the table name of the data table where the similar dataset is located, any second data field, and the second filename.

[0191] Label any second data field in the data table containing similar datasets to obtain the unlabeled data fields in each data table updated in this iteration process;

[0192] Mark the file corresponding to the second filename in at least one bucket to obtain the unmarked files in each bucket obtained from the update process of this round of iteration.

[0193] The data processing apparatus in this embodiment acquires multiple sets of first filenames and multiple sets of first datasets. For any set of first filenames, based on the filenames in each set, it determines a target dataset similar to any set of first filenames from the multiple first datasets. For any first data field in the target dataset, if a first filename in any set of first filenames matches any first data field, it generates first association lineage information based on the table name of the data table containing the target dataset, the first data field, and the first filename. This allows for similarity calculation of filenames of unstructured files clustered in a data lake and data fields in the same column or row of a structured data table to obtain association lineage information. This effectively establishes a mapping relationship between structured data tables and unstructured files, improving the effectiveness of querying or accessing unstructured files.

[0194] To achieve the above embodiments, this application also proposes an electronic device. Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device includes:

[0195] The memory 701, the processor 702, and the computer program stored on the memory 701 and executable on the processor 702.

[0196] When the processor 702 executes the program, it implements the data processing method provided in any of the above embodiments.

[0197] Furthermore, electronic devices also include:

[0198] Communication interface 703 is used for communication between memory 701 and processor 702.

[0199] The memory 701 is used to store computer programs that can run on the processor 702.

[0200] The memory 701 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0201] The processor 702 is used to implement the data processing method described in any of the above embodiments when executing the program.

[0202] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0203] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0204] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0205] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the data processing method provided in any of the above embodiments.

[0206] To implement the above embodiments, this application also proposes a computer program product that, when executed by an instruction processor, implements the data processing method provided in any of the above embodiments.

[0207] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0208] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0209] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0210] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0211] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0212] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0213] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0214] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A data processing method, characterized by, The method comprises: obtaining a plurality of first file name sets, wherein the first file name set comprises file names of files in the same cluster, and the cluster is obtained by clustering files in at least one storage bucket in a data lake; obtaining a plurality of first data sets, wherein the first data set comprises data fields in the same column or the same row in the same data table in the data lake; for any first file name set, determining a target data set similar to the first file name set from the plurality of first data sets according to the file names in the first file name set; for any first data field in the target data set, if there is a first file name in the first file name set that matches the first data field, generating first association blood relation information according to the table name of the data table where the target data set is located, the first data field and the first file name; the method comprises: for any first file name set, determining a first similarity between the first file name set and each first data set according to the file names in the first file name set; determining a candidate data set from each first data set according to the first similarity of each first data set, wherein the first similarity of the candidate data set is higher than a set similarity threshold; determining the target data set from each candidate data set.

2. The method of claim 1, wherein, the method comprises: for any candidate data set, determining the number of hits of each file name in the first file name set in the candidate data set; determining the target data set from each candidate data set according to the number of hits of each candidate data set; or, the candidate data set with the largest first similarity is taken as the target data set.

3. The method of claim 1, wherein, the method comprises: for any first file name set, extracting features of each file name in the first file name set to obtain first text features of the first file name set; extracting features of each data field in each first data set to obtain second text features of each first data set; determining the first similarity between the first file name set and each first data set according to the second similarity between the first text features and each second text feature.

4. The method of claim 3, wherein, the method comprises: for any first data set, sampling each data field in the first data set to obtain a sampled first data set, wherein the sampled first data set comprises a set number of data fields; Feature extraction is performed on each data field contained in any of the sampled first data sets to obtain second text features of the any of the first data sets.

5. The method of claim 1, wherein, The obtaining of the plurality of first data sets comprises: obtaining a plurality of initial data sets, wherein each of the initial data sets comprises all data fields in the same column or the same row in the same data table in the data lake; For any initial data set, a retention score of the any initial data set is determined according to a repetition rate of each data field in the any initial data set, wherein the retention score is in a negative correlation with the repetition rate; The first data sets are determined from the initial data sets according to the retention scores of the initial data sets.

6. The method of claim 1, wherein, The obtaining of the plurality of first file name sets comprises: For any storage bucket in the data lake, file information of each file in the any storage bucket is obtained; Each file in the any storage bucket is clustered according to the file information of each file in the any storage bucket to obtain at least one cluster; According to the file names of each file in the same cluster, one of the first file name sets is generated; The file information comprises at least one of the following: a file name, a file type and file metadata.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: According to the first association blood relationship information, the file metadata of the file corresponding to the first file name is updated.

8. The method according to any one of claims 1-6, characterized in that, The method further comprises: The any first data field in the data table where the target data set is located is marked; The file corresponding to the first file name in the at least one storage bucket is marked; According to the unmarked data fields in each data table and the unmarked files in each storage bucket, at least one iteration process is performed until a set stop iteration condition is met; The stop iteration condition comprises at least one of the following: The number of unmarked files in each storage bucket is less than a set first number threshold; The number of unmarked data fields in each first data set is less than a set second number threshold; The file name of each unmarked file does not match the unmarked data field in the first data set.

9. The method of claim 8, wherein, Any round of the iteration process comprises: At least one second file name set is obtained, wherein the second file name set comprises file names of each file in the same file set, and the file set is obtained by clustering the unmarked files in each storage bucket updated in the previous round of the iteration process; At least one second data set is obtained, wherein the second data set comprises unmarked data fields in the same column or the same row in the same data table updated in the previous round of the iteration process; For any second file name set, similar data sets are determined from each second data set according to each file name in the any second file name set; For any second data field in the similar data set, if there is a second file name in the any second file name set that matches the any second data field, second association blood relationship information is generated according to the table name of the data table where the similar data set is located, the any second data field and the second file name; marking the any second data field in the data table where the similar data set is located to obtain an unmarked data field in each data table updated by the iteration process in this round; marking the file corresponding to the second file name in the at least one storage bucket to obtain an unmarked file in each storage bucket updated by the iteration process in this round.

10. A data processing apparatus, characterized by, The apparatus comprises: a first obtaining module configured to obtain a plurality of first file name sets, wherein each first file name set includes file names of files in a same cluster, and the cluster is obtained by clustering files in at least one storage bucket in a data lake; a second obtaining module configured to obtain a plurality of first data sets, wherein each first data set includes data fields in a same column or a same row in a same data table in the data lake; a determining module configured to, for any first file name set, determine, according to file names in the any first file name set, a target data set similar to the any first file name set from the plurality of first data sets; a generating module configured to, for any first data field in the target data set, generate first association lineage information according to a table name of a data table where the target data set is located, the any first data field and a first file name in the any first file name set, in a case where the first file name exists in the any first file name set and matches the any first data field. The determining module is specifically configured to: determine, for any first file name set, a first similarity between the any first file name set and each first data set according to file names in the any first file name set; determine, according to the first similarity of each first data set, a candidate data set from each first data set, wherein the first similarity of the candidate data set is higher than a set similarity threshold; determine the target data set from each candidate data set.

11. An electronic device, comprising: comprise: a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the data processing method in any one of claims 1-9 when executing the program.

12. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the data processing method in any one of claims 1-9.

Images