A metadata classification method and apparatus

Abstract

The application discloses a metadata classification method and device, and the method comprises the following steps: obtaining to-be-classified data; inputting the to-be-classified data into a metadata classification model to obtain a classification result of the to-be-classified data, wherein the classification result is used for representing a mapping relationship between the to-be-classified data and a standard service metadata set; wherein the metadata classification model is obtained based on the standard service metadata set, the standard service metadata set is determined based on a matching relationship between initial metadata and a service metadata standard theme domain, the service metadata standard theme domain comprises metadata standards classified according to service types, and metadata in different standard service metadata sets corresponds to different service types. The application makes the data classification more efficient, and improves the accuracy and reliability of the data classification.

Description

Technical Field

[0001] This application relates to the field of data processing, specifically to a metadata classification method and apparatus. Background Technology

[0002] With the development of digitalization and informatization, enterprises possess a wide variety of data resources in large quantities. At the same time, as the scope of business and the scale of customers continue to expand, the scale of data generated by enterprises is also growing rapidly. However, the dispersed data elements, the huge amount of data, and the unstructured data types further complicate data management and increase the difficulty of data identification and classification.

[0003] In related technologies, data resources are classified manually. However, manual classification of data lacks unified data standards and complete classification methods, resulting in low classification efficiency and accuracy. Summary of the Invention

[0004] This application provides a metadata classification method and apparatus, which can improve the efficiency and accuracy of metadata classification.

[0005] In view of this, the first aspect of this application provides a metadata classification method, the method comprising:

[0006] Obtain the data to be classified;

[0007] The data to be classified is input into the metadata classification model to obtain the classification result of the data to be classified. The classification result is used to represent the mapping relationship between the data to be classified and the standard business metadata set.

[0008] The metadata classification model is trained based on the standard business metadata set, which is determined based on the matching relationship between the initial metadata and the standard subject domain of the business metadata. The standard subject domain of the business metadata includes metadata standards divided according to business type, and the metadata in different standard business metadata sets corresponds to different business types.

[0009] Optionally, the standard business metadata set is obtained in the following way:

[0010] Collect initial metadata;

[0011] Data element identification is performed on the initial metadata to obtain a data element set;

[0012] The data element set is matched with the business metadata standard subject domain to obtain the business type corresponding to each data element in the data element set;

[0013] The business type corresponding to the initial metadata is determined based on the business type corresponding to each data element in the data element set;

[0014] Based on the initial metadata corresponding to the same business type, determine the standard business metadata set corresponding to the business type.

[0015] Optionally, the step of performing data element identification on the initial metadata to obtain a data element set includes:

[0016] A first identification result is obtained by identifying data elements from the initial metadata using at least one of the following identification methods: basic identification, value range identification, and regular expression identification; wherein, the basic identification identifies data elements by field name, the value range identification identifies data elements by the value range of the data, and the regular expression identification identifies data elements by the regular expression rules of the data.

[0017] Based on the identification method corresponding to the data element in the first identification result, a corresponding weight is set for the data element to obtain a second identification result;

[0018] Based on the second identification result, a set of data elements is obtained.

[0019] Optionally, the method further includes:

[0020] Based on the mapping relationship, the data to be classified is matched with the standard resource directory to obtain the matching relationship between the data to be classified and the standard resource directory;

[0021] Based on the matching relationship, the data to be classified is managed in a catalog format.

[0022] Optionally, the step of cataloging the data to be classified according to the matching relationship includes:

[0023] If the matching relationship indicates that the data to be classified matches a standard resource directory, then the data to be classified is assigned to the standard resource directory;

[0024] If the matching relationship indicates that the data to be classified matches multiple standard resource directories, then a second matching is performed based on the number of fields in the data to be classified and the number of fields in the multiple standard resource directories, and the data to be classified is assigned to the standard resource directories that meet the preset conditions.

[0025] If the matching relationship indicates that the data to be classified does not match the standard resource directory, then the data elements in the data to be classified are filtered, and the standard resource directory matching is performed again.

[0026] Optionally, if the matching relationship indicates that the data to be classified does not match the standard resource directory, then filtering the data elements in the data to be classified and re-matching the standard resource directory includes:

[0027] When the matching relationship indicates that the data to be classified does not match the standard resource directory, data elements are identified from the data to be classified by at least one of the basic identification, value range identification and regular expression identification methods to obtain a third identification result.

[0028] Based on the identification method corresponding to the data element in the third identification result, a corresponding weight is set for the data element to obtain the fourth identification result;

[0029] Based on the fourth identification result, delete the data elements in the data to be classified that meet the weighted filtering conditions to obtain filtered data;

[0030] The filtered data is used to re-match the standard resource directory.

[0031] Optionally, the method further includes:

[0032] A standard technical metadata set is generated based on the standard business metadata set; the standard technical metadata set is used in the metadata classification model as supplementary information when the metadata classification model classifies the data to be classified.

[0033] A second aspect of this application provides a metadata classification apparatus, the apparatus comprising:

[0034] The data acquisition unit is used to: acquire data to be classified;

[0035] The data classification unit is used to: input the data to be classified into the metadata classification model to obtain the classification result of the data to be classified, wherein the classification result is used to represent the mapping relationship between the data to be classified and the standard business metadata set;

[0036] The metadata classification model is trained based on the standard business metadata set, which is determined based on the matching relationship between the initial metadata and the standard subject domain of the business metadata. The standard subject domain of the business metadata includes metadata standards divided according to business type, and the metadata in different standard business metadata sets corresponds to different business types.

[0037] Optionally, the apparatus further includes: a standard data generation unit, for:

[0038] Collect initial metadata;

[0039] Data element identification is performed on the initial metadata to obtain a data element set;

[0040] The data element set is matched with the business metadata standard subject domain to obtain the business type corresponding to each data element in the data element set;

[0041] The business type corresponding to the initial metadata is determined based on the business type corresponding to each data element in the data element set;

[0042] Based on the initial metadata corresponding to the same business type, determine the standard business metadata set corresponding to the business type.

[0043] Optionally, the standard data generation unit is specifically used for:

[0044] A first identification result is obtained by identifying data elements from the initial metadata using at least one of the following identification methods: basic identification, value range identification, and regular expression identification; wherein, the basic identification identifies data elements by field name, the value range identification identifies data elements by the value range of the data, and the regular expression identification identifies data elements by the regular expression rules of the data.

[0045] Based on the identification method corresponding to the data element in the first identification result, a corresponding weight is set for the data element to obtain a second identification result;

[0046] Based on the second identification result, a set of data elements is obtained.

[0047] As can be seen from the above technical solution, this application has the following advantages: This application obtains data to be classified; inputs the data to be classified into a metadata classification model to obtain the classification result of the data to be classified. The classification result is used to represent the mapping relationship between the data to be classified and the standard business metadata set. The metadata classification model is trained based on the standard business metadata set, which is determined based on the matching relationship between the initial metadata and the standard subject domain of the business metadata. The standard subject domain of the business metadata includes metadata standards divided according to business type, and the metadata in different standard business metadata sets corresponds to different businesses. By training the metadata classification model using a machine learning-based algorithm, the metadata classification of the data to be classified in the business system is completed, reducing manual operations, making data classification more efficient, and improving the accuracy and reliability of data classification. Attached Figure Description

[0048] Figure 1 A flowchart illustrating a metadata classification method provided in this application embodiment;

[0049] Figure 2 A flowchart illustrating a method for generating a standard business metadata set, provided in an embodiment of this application;

[0050] Figure 3 A flowchart illustrating a metadata classification method provided in an embodiment of this application scenario;

[0051] Figure 4 An architecture diagram of a metadata classification method provided for an embodiment of this application scenario;

[0052] Figure 5 This is a schematic diagram of a metadata classification device provided in an embodiment of this application. Detailed Implementation

[0053] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0054] See Figure 1 As shown in the figure, this application provides a metadata classification method, which specifically includes the following steps:

[0055] Step 101: Obtain the data to be classified.

[0056] When classifying metadata, the first step is to obtain the data to be classified through methods such as data collection. When the amount of collected metadata is large, the data to be classified can be a collection of multiple metadata entries. When the amount of collected metadata is small, the data to be classified must include at least one metadata entry. Metadata is descriptive data, which is structured data extracted from information resources to describe their characteristics and content (such as a book's title, edition, publication data, and related descriptions). It is used to organize, describe, retrieve, store, and manage information and data resources.

[0057] It should be noted that the information (including but not limited to electronic device information, account information, operation information, etc.), data (including but not limited to stored data, feature data, operation data, etc.) and signals involved in the embodiments of this application are all authorized by the relevant parties or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0058] Step 102: Input the data to be classified into the metadata classification model to obtain the classification result of the data to be classified. The classification result is used to represent the mapping relationship between the data to be classified and the standard business metadata set.

[0059] The metadata classification model is trained based on a standard business metadata set. This standard business metadata set is determined by the matching relationship between the initial metadata and the standard business metadata subject domains. These subject domains include metadata standards categorized by business type, with different standard business metadata sets corresponding to different business types. When classifying data to be classified, the metadata classification model maps the data to be classified to the data in the standard business metadata set, obtaining the mapping relationship between them, which is then represented by the classification result.

[0060] See Figure 2 As shown, in one possible implementation, the standard business metadata set is obtained in the following way:

[0061] Step 201: Collect initial metadata.

[0062] Initial metadata is obtained through data collection and other methods. Initial metadata can be a set of initial metadata that includes multiple metadata entries.

[0063] Step 202: Perform data element identification on the initial metadata to obtain a data element set.

[0064] Machine learning algorithms are used to match data elements, identifying them from the initial collected metadata and generating a set of data elements based on these identifications. A data element (also called a data unit) is a data unit described by a set of attributes, including its definition, identification, representation, and allowed values. In a given context, it is typically used to construct a semantically correct, independent, and unambiguous unit of information representing a specific concept. It is the smallest unit that makes up entity data, also known as atomic data or data element. For example, "Contact Person Name: Zhang San" in a customer contact information format can be understood as a data element, with "Contact Person Name" as the data element object and "Zhang San" as the data element value. A data element can be understood as the basic unit of data, a data unit / field used to store data; while metadata is data used to describe data, including data elements.

[0065] Specifically, step 202 can be achieved through the following steps:

[0066] Step 2021: Identify data elements from the initial metadata using at least one of the following identification methods: basic identification, range identification, and regularization identification, to obtain the first identification result.

[0067] Among them, basic recognition identifies data elements by field name, value range recognition identifies data elements by the value range of the data, and regular expression recognition identifies data elements by the regular expression rules of the data.

[0068] Basic recognition uses the attributes of the content carried in the field to match the attribute with the Chinese or English name of the field with the preset data elements already in the database. The attribute that successfully matches the preset data elements is extracted as the data element to obtain the recognition result. If the above recognition result is empty, synonyms can be queried according to the field name, and the data elements can be identified according to the synonyms of the field name to obtain the recognition result. The data type of the field in the recognition result is checked for consistency with the data type of the field in the data element, and the data elements with the same data type are retained.

[0069] Value range identification identifies the value range distribution range of data elements, determines the corresponding code dictionary based on the value range distribution range, and then determines the data elements to be used based on the mapping relationship between the code dictionary and the data elements.

[0070] Regular expression recognition includes ID card number recognition and mobile phone number recognition. It matches the regular expression rules of the content carried in the field with the regular expression rules of ID card number or mobile phone number. If the match is successful, the content carried in the field is extracted as a data element.

[0071] In one possible implementation, three identification methods can be used to identify data elements from the initial metadata in the order of basic identification, range identification, and regular expression identification. When the identification result obtained by basic identification is empty, range identification is used; when the identification result obtained by range identification is empty, regular expression identification is used.

[0072] In one possible implementation, three identification methods—basic identification, range identification, and regular expression identification—can be used simultaneously to identify data elements from the initial metadata.

[0073] Step 2022: Based on the recognition method corresponding to the data element in the first recognition result, set the corresponding weight for the data element to obtain the second recognition result.

[0074] By setting different confidence weights for the three recognition methods—basic recognition, range recognition, and regularization recognition—the confidence weights of the data elements obtained through different recognition methods will also be different. By mapping the data elements to their respective weight values, a second recognition result can be obtained.

[0075] Step 2023: Obtain the data element set based on the second identification result.

[0076] In the second identification result, each data element has its corresponding weight value. The data element set includes not only the individual data elements in the second identification result but also the weight values ​​corresponding to each data element. Furthermore, when generating the data element set from the second identification result, a confidence threshold can be set, and the data element set can be generated based on data elements whose weight values ​​are not less than the confidence threshold.

[0077] Data elements are identified from initial metadata using identification methods such as basic identification, value range identification, and regular expression identification. Different credibility weights are set for data elements obtained by different identification methods, making data element identification more efficient and improving the accuracy and credibility of data element identification.

[0078] Step 203: Match the data element set with the standard subject domain of the business metadata to obtain the business type corresponding to each data element in the data element set.

[0079] Within the business metadata standard subject area, different metadata standards are defined according to different business types. The data element set is matched against the business metadata standard subject area to obtain the correspondence between each data element in the data element set and its corresponding business type, thus determining the business type of the data element. For example, if the data element set contains a data element with the field name "Customer Name," and the business type of this data element in the business metadata standard subject area is "Financial Business," then based on the correspondence between "Customer Name" and "Financial Business," the business type corresponding to this data element and its corresponding metadata is determined to be "Financial Business."

[0080] Step 204: Determine the business type corresponding to the initial metadata based on the business type corresponding to each data element in the data element set.

[0081] Once the correspondence between data elements and business types is determined, the initial metadata corresponding to the data element is matched with the business type corresponding to the data element based on the correspondence between the data element and the initial metadata, thereby obtaining the correspondence between the initial metadata and business types.

[0082] Step 205: Based on the initial metadata corresponding to the same business type, determine the standard business metadata set corresponding to the business type.

[0083] In the business metadata standard subject area, different metadata standards are defined according to the business type. The initial metadata corresponding to the same business type is stored in the standard business metadata set corresponding to the business type according to the metadata standard corresponding to the business type. The metadata in different standard business metadata sets corresponds to different business types.

[0084] By identifying the initial metadata, a set of data elements is obtained. Based on the correspondence between the initial metadata, the set of data elements, and the standard subject areas of business metadata, the initial metadata is used to generate a set of standard business metadata according to the metadata standards corresponding to different business types, which is beneficial to the accuracy of metadata classification.

[0085] Furthermore, step 102 may be followed by the following steps:

[0086] Step 103: Based on the mapping relationship, match the data to be classified with the standard resource directory to obtain the matching relationship between the data to be classified and the standard resource directory.

[0087] The standard resource catalog contains data resource information for the entire industry; the classification results include the mapping relationship between the data to be classified and the standard business metadata set. When matching the standard resource catalog, the data to be classified can be matched to the industry catalog corresponding to the standard business metadata set based on the aforementioned mapping relationship.

[0088] Step 104: Based on the matching relationship, perform catalog management of the data to be classified.

[0089] Based on the matching relationship between the data to be classified and the standard resource directory, the data to be classified is placed in its corresponding industry directory, the information in the standard resource directory is updated, and the data to be classified is managed in a catalog manner.

[0090] Specifically, if the matching relationship indicates that the data to be classified matches a standard resource directory, then the data to be classified will be directly assigned to that standard resource directory.

[0091] If the matching relationship indicates that the data to be classified matches multiple standard resource directories, then a second matching is performed based on the number of fields in the data to be classified and the number of fields in the multiple standard resource directories, and the data to be classified is assigned to a standard resource directory that meets the preset conditions.

[0092] For example, when the data to be classified, A, matches two standard resource directories, namely directory B and directory C, then during the second matching, since the data to be classified, A, has a total of N fields (N is an integer greater than or equal to 1), and the corresponding number of fields in directory B and directory C are N+1 and 2N+1 respectively, since the difference between N+1 and N is less than the difference between 2N+1 and N, the data to be classified, A, is assigned to directory B.

[0093] If the matching relationship indicates that the data to be classified does not match the standard resource directory, then the data elements in the data to be classified are filtered, and the standard resource directory matching is performed again.

[0094] In one possible implementation, the data elements in the data to be classified can be filtered and matched using the following steps:

[0095] Step S1: When the matching relationship indicates that the data to be classified does not match the standard resource directory, the data elements are identified from the data to be classified by at least one of the following methods: basic identification, value range identification, and regular expression identification, to obtain the third identification result.

[0096] The process of identifying data elements from the data to be classified can refer to the content described in step 2021.

[0097] Step S2: Based on the recognition method corresponding to the data element in the third recognition result, weight the data elements to obtain the fourth recognition result.

[0098] The process of obtaining the fourth identification result based on the third identification result can refer to the content described in step 2022.

[0099] Step S3: Based on the fourth identification result, delete the data elements in the data to be classified that meet the weight filtering conditions to obtain the filtered data.

[0100] In the fourth identification result, each data element has its corresponding weight value. By using weight filtering conditions, data elements with lower weight values ​​are filtered out, while data elements with higher weight values ​​are retained to obtain filtered data.

[0101] Step S4: Re-match the standard resource directory using the filtered data.

[0102] Using filtered data to re-match the standard resource directory and lower the matching standard until the data to be classified matches at least one standard resource directory, is beneficial for achieving directory-based management of all the data to be classified mentioned above.

[0103] Based on the metadata standards in the standard business metadata set, standard resource catalog matching is performed, and the information under the standard resource catalog is improved accordingly. This enables catalog management of data resources, reduces the high time cost and low accuracy of manual classification, and improves the efficiency of metadata classification.

[0104] Furthermore, a standard technical metadata set can be generated based on the standard business metadata set, which can be used in the metadata classification model as supplementary information when the metadata classification model classifies the data to be classified.

[0105] The standard technical metadata set includes the English and Chinese names of fields, field descriptions, data types, value ranges, code encoding rules, and data formats of the metadata standards. During the training phase of the metadata classification model, incorporating the standard technical metadata set as supplementary information, and using its data structure and other information for further matching training, helps improve the accuracy of metadata classification.

[0106] This application embodiment obtains data to be classified; inputs the data to be classified into a metadata classification model to obtain a classification result, which represents the mapping relationship between the data to be classified and a standard business metadata set. The metadata classification model is trained based on the standard business metadata set, which is determined based on the matching relationship between initial metadata and standard business metadata subject domains. These standard business metadata subject domains include metadata standards categorized by business type, with metadata in different standard business metadata sets corresponding to different businesses. By training the metadata classification model using a machine learning-based algorithm, the metadata classification of data to be classified in the business system is completed, reducing manual operations, making data classification more efficient, and improving the accuracy and reliability of data classification.

[0107] like Figure 3 As shown in the example, the metadata classification method provided in this application includes the following steps:

[0108] Step 301: Collect initial metadata.

[0109] Initial metadata is collected through methods such as data acquisition. The initial metadata can be a set of initial metadata that includes multiple metadata entries.

[0110] Step 302: Identify data elements from the initial metadata.

[0111] Data elements are identified from the collected initial metadata using identification methods such as basic identification, value range identification, and regular expression identification.

[0112] Step 303: Weight the data element identification results to obtain the data element set.

[0113] The data elements identified in step 302 are weighted using machine learning algorithms, and a set of data elements is generated based on the weighted data elements.

[0114] Step 304: Match the data element set with the standard subject domain of the business data element to obtain the standard business metadata set.

[0115] The standard subject areas for business data elements in the credit system are shown below:

[0116]

[0117] It can be seen that the metadata standards corresponding to the subject "parties" are different in different business types. In some businesses, it may be "individual customers", "management information" or "customer identification".

[0118] The metadata standards for credit business in the standard business metadata set are as follows:

[0119]

[0120] Step 305: Generate a standard technical metadata set based on the standard business metadata set.

[0121] The standard technical metadata set corresponding to credit business is shown below:

[0122]

[0123]

[0124] Step 306: Train a metadata classification model based on the standard business metadata set and the standard technical metadata set.

[0125] A BiLSTM+CRF model, consisting of a Bi-directional Long Short-Term Memory (BiLSTM) network and a Conditional Random Field (CRF), is constructed for nonlinear mapping. The BiLSTM+CRF model comprises a look-up layer, a BiLSTM layer, and a CRF layer. Metadata is input to the look-up layer and represented as word vectors and character vector sequences. The BiLSTM layer then processes the input word vectors and character vector sequences through the Bi-directional Long Short-Term Memory neural network, outputting all metadata standard labels and their scores for each character. Finally, the CRF layer learns the dependencies between the metadata and outputs classification results that conform to the metadata standards.

[0126] Step 307: Input the dataset to be classified into the metadata classification model to obtain the classification results.

[0127] For example, the dataset to be classified is shown below:

[0128]

[0129]

[0130] Step 308: Match the classification results with the standard resource catalog to generate data resource catalog association relationships.

[0131] The data list in the standard resource directory corresponding to the credit system is as follows:

[0132]

[0133]

[0134] Furthermore, this application's scenario embodiment also provides an architecture diagram for a metadata classification method, such as... Figure 4 As shown, steps 301 to 308 are based on Figure 4 The method architecture diagram shown is executed.

[0135] See Figure 5 As shown in the figure, this application embodiment provides a metadata classification device, which includes: a data acquisition unit 501 and a data classification unit 502.

[0136] Data acquisition unit 501 is used for: acquiring data to be classified;

[0137] The data classification unit 502 is used to: input the data to be classified into the metadata classification model, obtain the classification result of the data to be classified, and the classification result is used to represent the mapping relationship between the data to be classified and the standard business metadata set;

[0138] The metadata classification model is trained based on a standard business metadata set. The standard business metadata set is determined based on the matching relationship between the initial metadata and the standard subject domain of the business metadata. The standard subject domain of the business metadata includes metadata standards divided by business type. The metadata in different standard business metadata sets corresponds to different business types.

[0139] Furthermore, Figure 4 The apparatus shown also includes a standard data generation unit for:

[0140] Collect initial metadata;

[0141] Data element identification is performed on the initial metadata to obtain a set of data elements;

[0142] Match the data element set with the standard subject area of ​​the business metadata to obtain the business type corresponding to each data element in the data element set;

[0143] Based on the business type corresponding to each data element in the data element set, determine the business type corresponding to the initial metadata;

[0144] Based on the initial metadata corresponding to the same business type, determine the standard business metadata set corresponding to the business type.

[0145] Furthermore, the standard data generation unit is specifically used for:

[0146] Data elements are identified from the initial metadata by at least one of the following identification methods: basic identification, value range identification, and regular expression identification, to obtain a first identification result; wherein, basic identification identifies data elements by field name, value range identification identifies data elements by the value range of the data, and regular expression identification identifies data elements by the regular expression rules of the data.

[0147] Based on the recognition method corresponding to the data element in the first recognition result, the corresponding weight is set for the data element to obtain the second recognition result;

[0148] Based on the second identification result, a set of data elements is obtained.

[0149] Furthermore, Figure 4 The device shown also includes a catalog management unit for:

[0150] Based on the mapping relationship, the data to be classified is matched with the standard resource directory to obtain the matching relationship between the data to be classified and the standard resource directory;

[0151] Based on the matching relationships, the data to be classified is managed in a catalog format.

[0152] Furthermore, the catalog management unit is specifically used for:

[0153] If the matching relationship indicates that the data to be classified matches a standard resource directory, then the data to be classified will be assigned to the standard resource directory;

[0154] If the matching relationship indicates that the data to be classified matches multiple standard resource directories, then a second matching is performed based on the number of fields in the data to be classified and the number of fields in the multiple standard resource directories, and the data to be classified is assigned to a standard resource directory that meets the preset conditions.

[0155] If the matching relationship indicates that the data to be classified does not match the standard resource directory, then the data elements in the data to be classified are filtered, and the standard resource directory matching is performed again.

[0156] Furthermore, the catalog management unit is specifically used for:

[0157] When the matching relationship indicates that the data to be classified does not match the standard resource directory, data elements are identified from the data to be classified by at least one of the basic identification, value range identification and regularity identification methods to obtain a third identification result.

[0158] Based on the recognition method corresponding to the data element in the third recognition result, set the corresponding weight for the data element to obtain the fourth recognition result;

[0159] Based on the fourth identification result, delete the data elements in the data to be classified that meet the weight filtering conditions to obtain the filtered data;

[0160] Use the filtered data to re-match the standard resource catalog.

[0161] Furthermore, the standard data generation unit is also used for:

[0162] A standard technical metadata set is generated based on the standard business metadata set. The standard technical metadata set is used in the metadata classification model as supplementary information when the metadata classification model classifies the data to be classified.

[0163] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0164] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.

[0165] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0166] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0167] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0168] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0169] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0171] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0172] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A metadata classification method, characterized in that, The method includes: Obtain the data to be classified; The data to be classified is input into the metadata classification model to obtain the classification result of the data to be classified. The classification result is used to represent the mapping relationship between the data to be classified and the standard business metadata set. The metadata classification model is trained based on the standard business metadata set, which is determined based on the matching relationship between the initial metadata and the standard subject domain of the business metadata. The standard subject domain of the business metadata includes metadata standards divided according to business type, and the metadata in different standard business metadata sets corresponds to different business types. The standard business metadata set is obtained through the following methods: Collect initial metadata; Data element identification is performed on the initial metadata to obtain a data element set; The data element set is matched with the business metadata standard subject domain to obtain the business type corresponding to each data element in the data element set; The business type corresponding to the initial metadata is determined based on the business type corresponding to each data element in the data element set; Initial metadata corresponding to the same business type is stored in the standard business metadata set corresponding to the business type, according to the metadata standard corresponding to the business type. The step of identifying data elements from the initial metadata to obtain a set of data elements includes: Simultaneously, basic identification, value range identification, and regular expression identification are used to identify data elements from the initial metadata; wherein, at least one of the basic identification, value range identification, and regular expression identification methods is used to identify data elements from the initial metadata to obtain a first identification result; wherein, the basic identification identifies data elements by field name, the value range identification identifies data elements by the value range of the data, and the regular expression identification identifies data elements by the regular expression rules of the data, and different identification methods correspond to different confidence weights; Based on the identification method corresponding to the data element in the first identification result, a corresponding weight is set for the data element to obtain a second identification result; A set of data elements is generated based on the data elements in the second identification result whose weight values ​​are not less than the confidence threshold. Based on the mapping relationship, the data to be classified is matched with the standard resource directory to obtain the matching relationship between the data to be classified and the standard resource directory; wherein, the standard resource directory contains data resource information of the entire industry, and based on the mapping relationship, the data to be classified is matched to the industry directory corresponding to the standard business metadata set; Based on the matching relationship, the data to be classified is managed in a catalog format; When the matching relationship indicates that the data to be classified does not match the standard resource directory, data elements are identified from the data to be classified by at least one of the basic identification, value range identification and regular expression identification methods to obtain a third identification result. Based on the identification method corresponding to the data element in the third identification result, a corresponding weight is set for the data element to obtain the fourth identification result; Based on the fourth identification result, data elements that meet the weight filtering conditions in the data to be classified are deleted to obtain filtered data; wherein, in the fourth identification result, each data element has its corresponding weight value, and the weight filtering conditions are used to filter out data elements with lower weight values ​​and retain data elements with higher weight values. The filtered data is used to re-match the standard resource directory, lowering the matching standard until the data to be classified matches at least one standard resource directory.

2. The method according to claim 1, characterized in that, The step of cataloging the data to be classified based on the matching relationship includes: If the matching relationship indicates that the data to be classified matches a standard resource directory, then the data to be classified is assigned to the standard resource directory; If the matching relationship indicates that the data to be classified matches multiple standard resource directories, then a second matching is performed based on the number of fields in the data to be classified and the number of fields in the multiple standard resource directories, and the data to be classified is assigned to the standard resource directories that meet the preset conditions. If the matching relationship indicates that the data to be classified does not match the standard resource directory, then the data elements in the data to be classified are filtered, and the standard resource directory matching is performed again.

3. The method according to claim 1, characterized in that, The method further includes: A standard technical metadata set is generated based on the standard business metadata set; the standard technical metadata set is used in the metadata classification model as supplementary information when the metadata classification model classifies the data to be classified.

4. A metadata classification device, characterized in that, The device includes: The data acquisition unit is used to: acquire data to be classified; The data classification unit is used to: input the data to be classified into the metadata classification model to obtain the classification result of the data to be classified, wherein the classification result is used to represent the mapping relationship between the data to be classified and the standard business metadata set; The metadata classification model is trained based on the standard business metadata set, which is determined based on the matching relationship between the initial metadata and the standard subject domain of the business metadata. The standard subject domain of the business metadata includes metadata standards divided according to business type, and the metadata in different standard business metadata sets corresponds to different business types. The apparatus further includes: a standard data generation unit, used for: Collect initial metadata; Data element identification is performed on the initial metadata to obtain a data element set; The data element set is matched with the business metadata standard subject domain to obtain the business type corresponding to each data element in the data element set; The business type corresponding to the initial metadata is determined based on the business type corresponding to each data element in the data element set; The initial metadata corresponding to the same business type is stored in the standard business metadata set corresponding to the business type according to the metadata standard corresponding to the business type; the standard data generation unit is specifically used for: Simultaneously, basic identification, value range identification, and regular expression identification are used to identify data elements from the initial metadata; wherein, at least one of the basic identification, value range identification, and regular expression identification methods is used to identify data elements from the initial metadata to obtain a first identification result; wherein, the basic identification identifies data elements by field name, the value range identification identifies data elements by the value range of the data, and the regular expression identification identifies data elements by the regular expression rules of the data, and different identification methods correspond to different confidence weights; Based on the identification method corresponding to the data element in the first identification result, a corresponding weight is set for the data element to obtain a second identification result; A set of data elements is generated based on the data elements in the second identification result whose weight values ​​are not less than the confidence threshold. The device further includes a catalog management unit for: Based on the mapping relationship, the data to be classified is matched with the standard resource directory to obtain the matching relationship between the data to be classified and the standard resource directory; wherein, the standard resource directory contains data resource information of the entire industry, and according to the mapping relationship, the data to be classified is matched to the industry directory corresponding to the standard business metadata set. Based on the matching relationships, the data to be classified is managed in a catalog format. When the matching relationship indicates that the data to be classified does not match the standard resource directory, data elements are identified from the data to be classified by at least one of the basic identification, value range identification and regular expression identification methods to obtain a third identification result. Based on the identification method corresponding to the data element in the third identification result, a corresponding weight is set for the data element to obtain the fourth identification result; Based on the fourth identification result, data elements that meet the weight filtering conditions in the data to be classified are deleted to obtain filtered data; wherein, in the fourth identification result, each data element has its corresponding weight value, and the weight filtering conditions are used to filter out data elements with lower weight values ​​and retain data elements with higher weight values. The filtered data is used to re-match the standard resource directory, lowering the matching standard until the data to be classified matches at least one standard resource directory.

Images