A metadata classification method based on automatic identification technology, a storage medium and a system

Abstract

The application provides a metadata classification method, a storage medium and a system based on automatic identification technology, which comprises dividing sensitive types of field metadata and dividing subjects of table metadata. In the process of dividing sensitive types of field metadata and dividing subjects of table metadata, business personnel only need to divide sensitive types of part of the field metadata and divide subjects of part of the table metadata, and then the system can automatically divide sensitive types of the remaining field metadata based on a trained binary classifier model and automatically divide subjects of the remaining table metadata based on a trained word vector model. Compared with the traditional metadata classification method in which business personnel manually classify and sort metadata throughout the whole process, the method saves time and effort.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a metadata classification method based on automatic identification technology, a storage medium and a system. BACKGROUND

[0002] Under the background of deepening digital transformation, data has become a core element to enhance the competitiveness of enterprises. Metadata is data used to describe data in an enterprise, which is equivalent to the DNA of enterprise data. It describes the information of data attributes and reflects the data transactions, practices, objects and relationships. Metadata effectively helps technical developers and business personnel improve work efficiency, shorten the development cycle and ensure the efficient and accurate operation of various IT systems by describing the source, storage, usage, business meaning, conversion rules, data change management and data management personnel and other related information.

[0003] In the digital age, how to strengthen data management and release the value of data elements has become a basic work that all walks of life highly value. Improving data management capability has become the top priority to promote enterprise digital transformation and promote the development of digital economy. Enterprises want to know the quality of data in their hands, the life cycle of data, which data needs to be protected in terms of security and privacy, and who can use these data, how to use them… These problems need to be solved through metadata management. If there is no effective metadata management, the data assets of the enterprise may become a burden for the enterprise to increase profits. Metadata classification is a basic component of metadata management. In the case of relevant data in an enterprise, the metadata content is sorted and classified to better assist metadata management related work. Metadata classification generally needs to divide the sensitive types of field metadata and the themes of table metadata. However, since field metadata is generally complex and has no uniform naming rules, and the content of table metadata is generally chaotic, it is difficult to accurately classify using an automated metadata classification method. Therefore, the traditional metadata classification method generally requires business personnel to manually classify and sort table metadata and field metadata, which is time-consuming and laborious. SUMMARY

[0004] The technical problem to be solved by the present application is how to reduce the manual work and time consumption of metadata classification.

[0005] To solve the above technical problems, the present application provides a metadata classification method based on automatic identification technology, comprising the following steps:

[0006] A. dividing the sensitive types of field metadata, specifically comprising the following steps A1-A5;

[0007] A1. selecting field metadata of a sensitive type to be divided from a plurality of table metadata stored in a data warehouse, and finding a plurality of specific information associated with the field metadata from the plurality of table metadata stored in the data warehouse according to the field metadata;

[0008] A2. obtaining a result of a sensitive type division of part of the specific information of the field metadata by a business staff according to business experience and / or business rules, the result including sensitive and insensitive;

[0009] A3. dividing the part of the specific information of the sensitive type into a first training set data and a first test set data according to a preset proportion, and then performing a text preprocessing operation on each first training set data, the text preprocessing operation including a word segmentation operation and a stop word removal operation;

[0010] A4. calculating a feature weight of each first training set data and each first test set data, generating a training input vector according to the feature weight of each first training set data, generating a test input vector according to the feature weight of each first test set data, then inputting each training input vector and each test input vector into a binary classifier model based on a support vector machine (SVM) to perform training, obtaining a trained binary classifier model, and then using the trained binary classifier model to divide the remaining specific information of the field metadata into a sensitive type, the division result including sensitive and insensitive;

[0011] A5. obtaining a sensitive type of each division of all specific information of the field metadata, counting a proportion of specific information of a sensitive type, if the proportion is not less than a preset threshold, dividing the field metadata into a sensitive field, if the proportion is less than the preset threshold, dividing the field metadata into an insensitive field;

[0012] B. dividing a theme of table metadata, specifically including the following steps B1-B5;

[0013] B1. obtaining a plurality of table metadata from a data warehouse, and obtaining a result of a theme division of part of the table metadata by a business staff according to business application classification;

[0014] B2. dividing the part of the table metadata of the divided theme into a second training set data and a second test set data according to a preset proportion, then performing a text preprocessing operation on each second training set data, the text preprocessing operation including a string merging, a word segmentation operation and a stop word removal operation, and then inputting each second training set data and each second test set data into a word vector model based on a word2vec corpus to perform training, obtaining a trained word vector model;

[0015] B3. Obtain a plurality of preset subject domains, and obtain representative table metadata selected by a business staff from the data warehouse according to business experience for each subject domain, then calculate the feature weight of each word in each representative table metadata, select a plurality of words with a feature weight ranking in a preset position as the central word of the subject domain to which the representative table metadata belongs and which needs to be vectorized, and then input each central word into the trained word vector model to obtain the vectorized representation of each central word, thereby obtaining a plurality of central word vectors;

[0016] B4. Perform a text preprocessing operation on the remaining table metadata, the text preprocessing operation including a string merging, a word segmentation operation and a stop word removal operation, thereby obtaining a string word segmentation result of the remaining table metadata;

[0017] B5. For each remaining table metadata, calculate the cosine similarity between the string word segmentation result and each central word vector, obtain the maximum cosine similarity, and then obtain the subject domain to which the central word vector corresponding to the maximum cosine similarity belongs, and divide the remaining table metadata into the subject domain.

[0018] Preferably, in step A4, the feature weight is calculated as follows: obtaining a plurality of words obtained after the word segmentation operation, calculating the TFIDF value of each word, and the specific formula is as follows:

[0019] TFIDF = TF x log(n / (m+1));

[0020] Wherein, TF represents the word frequency of the word, n represents the total number of specific information under the field metadata, and m represents the number of specific information containing the word under the segment metadata.

[0021] Preferably, in step A4, the SVM support vector machine-based binary classifier model is trained as follows: the k-fold cross-validation method is used to find the optimal parameters of the SVM support vector machine-based binary classifier model, and then the optimal parameters are used to reset the SVM support vector machine-based binary classifier model to obtain the trained binary classifier model.

[0022] Preferably, in step A4, the k-fold cross-validation method is used to find the optimal parameters of the SVM support vector machine-based binary classifier model, which specifically includes:

[0023] First, determine the parameters that need to be selected for optimal parameters as the penalty term coefficient C of the relaxation coefficient, the coefficient gamma of the kernel function and the selection kernel of the kernel function, and then randomly generate a set {C, gamma, kernel} as the initial position of the particle according to the value range of these parameters;

[0024] Second step: divide the training input vector into k subsets S1, S2, …, Sk evenly;

[0025] Third step: initialize e = 1;

[0026] Fourth step: leave a subset Se of the training input vector as the test set, and combine the rest of the subsets as the training set, and use the training set to train the binary classifier model;

[0027] Fifth step: test the binary classifier model using the test input vector to obtain the comprehensive performance evaluation index value f e of the Se subset;

[0028] Sixth step: e = e + 1;

[0029] Seventh step: repeat the fourth and fifth steps until e = k, and record the comprehensive performance evaluation index value f e corresponding to each e;

[0030] Eighth step: calculate the generalization comprehensive evaluation score F of this set of {C, gamma, kernel}, which is the average of the comprehensive performance evaluation index values f e corresponding to each e, and the specific formula is as follows:

[0031]

[0032] Ninth step: repeat the first to eighth steps until all combinations of {C, gamma, kernel} are traversed;

[0033] Tenth step: select the set of {C, gamma, kernel} with the maximum generalization comprehensive evaluation score F as the optimal parameters of the binary classifier model.

[0034] Preferably, the formula of the comprehensive performance evaluation index value f e is defined as:

[0035] where Precision is the accuracy rate, and its formula is defined as Recall is the recall rate, and its formula is defined as where TP represents the number of correctly classified metadata sensitive types, FP represents the number of metadata sensitive types that are not sensitive but are incorrectly predicted as sensitive, and FN represents the number of metadata sensitive types that are sensitive but are incorrectly predicted as not sensitive.

[0036] Preferably, in step B3, the top 20 feature weight sizes are selected as the central words that need to be vectorized in the subject domain to which the representative table metadata belongs.

[0037] Preferably, in the step B5, the cosine similarity calculation formula is as follows:

[0038]

[0039] Wherein, VV i represents the central word vector set under each subject domain, i represents the item number of one of the plurality of subject domains; vt N represents the feature word vector set of the remaining table metadata, and N represents the item number of the remaining table metadata to be divided into subject domains.

[0040] Preferably, in the step A3 and the step B2, the preset ratio is 7:3.

[0041] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps in the metadata classification method.

[0042] The application further provides a metadata classification system based on automatic identification technology, which comprises a computer readable storage medium and a processor connected with each other, and the computer readable storage medium is as described above.

[0043] The present application has the following beneficial effects: when the present application divides the sensitive type of field metadata, the result of the sensitive type division of the part of the specific information of the field metadata by the business personnel is obtained first, then the binary classifier model is trained based on the part of the specific information, and the trained binary classifier model is used to automatically recognize the sensitive type and divide the remaining specific information of the field metadata, and then the field metadata is divided into sensitive field or insensitive field based on the proportion of the specific information that is sensitive to the sensitive type; when the subject of table metadata is divided, the result of the subject division of part of the table metadata by the business personnel is obtained first, then the word vector model is trained based on the part of the table metadata, then the representative table metadata selected by the business personnel for each subject domain is obtained, then the center word that needs to be vectorized in the subject domain to which each representative table metadata belongs is obtained, then each center word is input into the trained word vector model to automatically recognize a plurality of center word vectors, then the string segmentation result of the remaining table metadata is obtained, then the cosine similarity between each string segmentation result of the remaining table metadata and each center word vector is calculated, the subject domain to which the center word vector corresponding to the maximum cosine similarity belongs is obtained, and the remaining table metadata is divided in the subject domain. In this way, when the metadata is classified, the business personnel only needs to divide the sensitive type of part of the field metadata and divide the subject of part of the table metadata, and then the system can automatically divide the sensitive type of the remaining field metadata based on the trained binary classifier model and automatically divide the subject of the remaining table metadata based on the trained word vector model. Compared with the traditional metadata classification method in which the business personnel manually classifies and sorts the metadata throughout the process, the present application saves time and effort. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is a flowchart of the metadata classification method based on automatic recognition technology.

[0045] Figure 2 It is a sample data representation of specific information.

[0046] Figure 3 It is a schematic diagram of the feature weight table of field specific information.

[0047] Figure 4 It is a schematic diagram of the feature weight table after the TFIDF value is input.

[0048] Figure 5 It is a sample data representation of table metadata. DETAILED DESCRIPTION

[0049] The present application will be further described in detail in combination with the specific embodiments.

[0050] The embodiment provides a metadata classification system based on automatic identification technology, which comprises a computer readable storage medium and a processor connected to each other, and a computer program is stored on the computer readable storage medium and is executed by the processor to realize a metadata classification method based on automatic identification technology as shown in the following formula (1). Figure 1 The metadata classification method based on automatic identification technology comprises the following steps A and B.

[0051] A. The sensitive type of field metadata is divided, and the sensitive type of field metadata specifically comprises the following steps A1-A5.

[0052] Enterprise metadata is generally stored in a data warehouse, and the metadata includes field metadata and specific information of the field metadata which needs to be protected in security and privacy. The field metadata which needs to be protected in security and privacy is a sensitive field, and the specific information of the field metadata which needs to be protected in security and privacy is sensitive information. When the metadata is managed, the data warehouse needs to be convenient for users to use, and the sensitive field and the sensitive information stored in the data warehouse need to be protected from being accessed by others at will. For this purpose, on the one hand, it is necessary to know which field metadata in the data warehouse belongs to the sensitive field, and it is necessary to know which specific information of the field metadata belongs to the sensitive information. On the other hand, the sensitive field and the sensitive information of the field metadata in the data warehouse need to be managed in a hierarchical permission. In the embodiment, a binary classifier model based on a support vector machine (SVM) is used to divide the sensitive type of the specific information of the field metadata, so as to obtain the sensitive information, and the sensitive type of the field metadata is divided according to the proportion of the sensitive information, so as to obtain the sensitive field. Then, the sensitive information and the sensitive field are managed in a hierarchical permission, and the specific steps A1-A5 are as follows.

[0053] A1. The field metadata which needs to be divided in the sensitive type is selected from the plurality of table metadata stored in the data warehouse, and the plurality of specific information associated with the field metadata is found from the plurality of table metadata stored in the data warehouse.

[0054] In the data warehouse, the multiple field metadata are stored in the form of tables, forming multiple table metadata. Therefore, when dividing the sensitive types of the field metadata, the system first identifies the multiple table metadata stored in the data warehouse, and then selects the field metadata that needs to be divided into sensitive types from the multiple table metadata according to the input information of the user. The selected field metadata is text data, including: name, residential address, IP address, family information, religious belief, gene, personal health condition, occupation, work unit, native place, interest, industry, short message and multimedia message content, email content involved in mobile internet services, group publishing content, chat record, content accessed by the user on the internet, etc. Then the system finds out the specific information of the related fields in the data warehouse according to the selected field metadata. For example, if the selected field metadata is name, the specific information found according to the field metadata is Zhang San, Li Si, Wang San San, Xiao Chen, Liu Zong…; or if the selected field metadata is work unit, the specific information found according to the field metadata is the personnel department of the tax bureau of city A, the risk management department of the bank of city B, the unified department of the medical insurance service hall of city C, the technology company of city D, the mechanical manufacturing company of city E…; or if the selected field metadata is residential address, the specific information found according to the field metadata is a road of b district of city A, c road of d district of city B, e road of f district of city C, g district of city D, h road of city E…

[0055] A2. Obtain the result of the division of the sensitive types of the part of the specific information of the field metadata by the business personnel according to the business experience and / or business rules, which includes sensitive and non-sensitive.

[0056] After finding the multiple specific information of the field metadata, the business personnel can divide the part of the specific information of each field metadata into sensitive type according to business experience and / or business rules, for example, the system finds 500 specific information according to the field metadata of "name", the business personnel can divide 50 specific information into sensitive type according to business experience and / or business rules, and the remaining 450 specific information is not divided into sensitive type, specifically, Zhang San, Li Si, Wang San San… These specific information are complete and detailed name information, and the business personnel can divide the sensitive type into sensitive, while Xiao Chen, Liu Zong… These specific information are not complete and detailed name information, and the business personnel can divide the sensitive type into insensitive. For another example, the system finds 500 specific information according to the field metadata of "work unit", the business personnel can divide 50 specific information into sensitive type according to business experience and / or business rules, specifically, the personnel department of A city tax bureau, risk management department of B city bank, unified department of C city medical insurance service hall… These specific information are complete and detailed work unit information, and the business personnel can divide the sensitive type into sensitive, while D city science and technology company, E city mechanical manufacturing company… These specific information are not complete and detailed work unit information, and the business personnel can divide the sensitive type into insensitive. For another example, the system finds 500 specific information according to the field metadata of "residence address", the business personnel can divide 50 specific information into sensitive type according to business experience and / or business rules, specifically, A district B road of A city, C district D road of B city, E district F road of C city… These specific information are complete and detailed residence address information, and the business personnel can divide the sensitive type into sensitive, while G district of D city, H road of E city… These specific information are not complete and detailed residence address information, and the business personnel can divide the sensitive type into insensitive. Then the system obtains the result of the business personnel dividing the part of the specific information of the field metadata into sensitive type.

[0057] A3. Divide the part of the specific information of the divided sensitive type into the first training set data and the first test set data according to the preset proportion, and then perform text preprocessing operation on each first training set data and each first test set data, the text preprocessing operation includes word segmentation operation and stop word removal operation.

[0058] After the system obtains the result of the business personnel dividing the part of the specific information of the field metadata into sensitive type, for example, after obtaining the result of the business personnel dividing 50 specific information under the field metadata "name" into sensitive type, according to the part of the specific information of the divided sensitive type, a sample data table of specific information as shown in Figure 2 is constructed, and then the system uses non-replacement random sampling method to divide the sample data table of specific information into the first training set data and the first test set data according to the proportion of 7:3. Figure 2The sample data shown in the sample data table is randomly selected 70% sample data divided into a first training set data, and the remaining 30% sample data is divided into a first test set data. Then the system performs text preprocessing operation on each first training set data and each first test set data, and the text preprocessing operation process includes the following two operations:

[0059] (1) Jieba word segmentation operation. Jieba is an excellent Chinese word segmentation third-party library, which uses a Chinese word library to determine the correlation probability between Chinese characters. The word group with high correlation probability between Chinese characters forms the segmentation result. There are three modes for jieba word segmentation operation: the first is the accurate mode, in which the text is accurately divided, and there is no redundant word; the second is the full mode, in which all possible words in the text are scanned out, and there is redundancy (i.e. a text can be cut into different words from different angles), in which different words can be mined out; the third is the search engine mode, which further divides long words based on the accurate mode. The jieba word segmentation operation of the present embodiment adopts the accurate mode.

[0060] (2) Remove stop words operation. Stop words are functional words in human language, which are generally functional and have no actual meaning. Removing them can save storage space and improve search efficiency. In this paper, the Harbin Institute of Technology stop word list is used to remove stop words from the first training set data.

[0061] For example, two first training set data are "Zhang San" and "Wang San San", then after jieba word segmentation operation on "Zhang San", two words "Zhang" and "San" are obtained, then remove stop words operation is performed on "Zhang" and "San" respectively, it is found that neither "Zhang" nor "San" contains stop words, so there is no need to remove stop words, and the original first training set data "Zhang San" is maintained; after jieba word segmentation operation on "Wang San San", three words "Wang", "San" and "San" are obtained, then remove stop words operation is performed on "Wang", "San" and "San" respectively, it is found that neither "Wang", "San" nor "San" contains stop words, so there is no need to remove stop words, and the original first training set data "Wang San San" is maintained.

[0062] In other embodiments, assuming that a certain first training set data is "the risk management department of City B Bank", four words can be obtained after performing the jieba word segmentation operation on "the risk management department of City B Bank", which are "City B", "Bank", "of", and "risk management department". Then, the stop words removal operation is performed on "City B", "Bank", "of", and "risk management department", respectively. It is found that one of the words "of" is a stop word in the Harbin Institute of Technology stop word table, so the stop word "of" is removed, leaving "City B", "Bank", and "risk management department". Therefore, the first training set data becomes "City B Bank risk management department".

[0063] It should be noted that the text preprocessing operation process of the first test set data is the same as that of the first training set data, which will not be described here.

[0064] A4. Calculate the feature weights of each first training set data and each first test set data, generate training input vectors according to the feature weights of each first training set data, generate test input vectors according to the feature weights of each first test set data, then input each training input vector and each test input vector into the SVM support vector machine-based binary classifier model for training, obtain the trained binary classifier model, and then use the trained binary classifier model to perform sensitive type division on the remaining specific information of the field metadata. The division result includes sensitive and insensitive.

[0065] After performing the text preprocessing operation on each first training set data and each first test set data, the feature weight table as shown in Figure 3 is constructed according to the word segmentation result, and then the system calculates the feature weights of each first training set data and each first test set data. The calculation object is the TFIDF value of each word. Then, the numerical value of each word in the feature weight table is recorded as the corresponding TFIDF value. Then, the input vectors required for training the SVM support vector machine-based binary classifier model are generated according to the feature weights of each specific information.

[0066] Taking the calculation of the feature weight of a certain first training set data as an example, the calculation object is the TFIDF value of each word in the first training set data. For a certain word, the word frequency TF of the word in the first training set data, the total number n of specific information under the field metadata to which the first training set data belongs, and the number m of specific information containing the word under the field metadata to which the first training set data belongs are first counted. The word frequency TF refers to the frequency of a certain word appearing in the first training set data, and the value is specifically the ratio of the frequency of a certain word appearing in the first training set data to the total frequency of the word in the first training set data. For example, in the first training set data "Zhang San", there are two words "Zhang" and "San" according to the above, and the total frequency of the word is 2, and the frequency of the word "Zhang" is 1, and the frequency of the word "San" is 1, so the word frequency TF of the word "Zhang" is 1 / 2, and the word frequency TF of the word "San" is 1 / 2. For another example, in the first training set data "Wang San San", there are three words "Wang", "San" and "San" according to the above, and the total frequency of the word is 3, and the frequency of the word "Wang" is 1, and the frequency of the word "San" is 2, so the word frequency TF of the word "Wang" is 1 / 3, and the word frequency TF of the word "San" is 2 / 3.

[0067] Then the TF-IDF method is used to calculate the TFIDF value of each word in the first training set data, and the calculation formula of the TF-IDF method is specifically: TFIDF = TF x log(n / (m+1)).

[0068] For example, for the first training set data "Zhang San", the field metadata to which it belongs is "name", and it is counted that there are 10,000 specific information under the field metadata "name", so n is 10,000; when calculating the TFIDF value of the word "Zhang", it is counted that there are 1,000 name information under the field metadata "name" containing the word "Zhang", so m is 1,000; and when calculating the TFIDF value of the word "San", it is counted that there are 400 name information under the field metadata "name" containing the word "San", so m is 400. Therefore, the TFIDF value of the word "Zhang" in the first training set data "Zhang San" is calculated by the TF-IDF method to obtain TFIDF = TF x log(n / (m+1)) = 1 / 2 x log(10000 / (1000+1)) ≈ 0.5, and the TFIDF value of the word "San" in the first training set data "Zhang San" is calculated by the same method to obtain TFIDF = TF x log(n / (m+1)) = 1 / 2 x log(10000 / (400+1)) ≈ 0.7. In addition to "Zhang" and "San", the first training set data "Zhang San" does not contain other words, so the TFIDF value corresponding to other words is 0.

[0069] For example, for the first training set data "Wang San San", the field metadata to which it belongs is "name", and statistics show that there are 10,000 specific information under the field metadata "name", so n is 10,000; when calculating the weight of the word "Wang", statistics show that there are 500 name information under the field metadata "name" containing the word "Wang", so m is 500; and when calculating the weight of the word "San", statistics show that there are 400 name information under the field metadata "name" containing the word "San", so m is 400. Therefore, the TFIDF value of the word "Wang" in the first training set data "Wang San San" is calculated by using the TF-IDF method, and TFIDF = TF x log(n / (m+1)) = 1 / 3 x log(10000 / (500+1)) = 0.43, and the TFIDF value of the word "San" in the first training set data "Wang San San" is calculated by using the TF-IDF method, and TFIDF = TF x log(n / (m+1)) = 2 / 3 x log(10000 / (400+1)) = 0.93. In addition to "Wang" and "San", the first training set data "Wang San San" does not contain other words, so the weight of other words is 0.

[0070] It should be noted that the feature weight calculation process of the first test set data and the generation process of the test input vector are the same as the first training set data. For example, one of the first test set data is "Li Si", then according to step A3 above, there are two words "Li" and "Si" in the first test set data "Li Si", so the total frequency of the words is 2, the frequency of the word "Li" is 1, and the frequency of the word "Si" is 1, so the word frequency TF of the word "Li" is 1 / 2, and the word frequency TF of the word "Si" is 1 / 2, and the field metadata to which the first test set data "Li Si" belongs is "name", and statistics show that there are 10,000 specific information under the field metadata "name", so n is 10,000; when calculating the TFIDF value of the word "Li", statistics show that there are 800 name information under the field metadata "name" containing the word "Li", so m is 800, and when calculating the TFIDF value of the word "Si", statistics show that there are 300 name information under the field metadata "name" containing the word "Si", so m is 300; therefore, the TFIDF value of the word "Li" in the first test set data "Li Si" is calculated by using the TF-IDF method, and TFIDF = TF x log(n / (m+1)) = 1 / 2 x log(10000 / (800+1)) = 0.55, and the TFIDF value of the word "Si" in the first test set data "Li Si" is calculated by using the TF-IDF method, and TFIDF = TF x log(n / (m+1)) = 1 / 2 x log(10000 / (300+1)) = 0.76. In addition to "Li" and "Si", the first test set data "Li Si" does not contain other words, so the TFIDF value of other words is 0.

[0071] This embodiment uses the first training set data "Zhang San" and "Wang San San" and the first test set data "Li Si" as examples. Figure 3 If the word values ​​in the feature weight table shown are entered as the corresponding TFIDF values, then the feature weight table becomes as follows: Figure 4 As shown. By Figure 4 Then, the corresponding training input vector (0.5, 0.7, 0, ..., 0) can be generated based on the feature weights of the first training set data "Zhang San", the corresponding training input vector (0, 0.93, 0.43, 0, ..., 0) can be generated based on the feature weights of the first training set data "Wang San San", and the corresponding test input vector (0, 0, 0, 0.5, 0.76, 0, ..., 0) can be generated based on the feature weights of the first test set data "Li Si".

[0072] After generating corresponding training and test input vectors based on the feature weights of each first training set and each first test set, the system inputs these vectors into a binary classifier model based on Support Vector Machine (SVM) for training. SVM is a statistical machine learning model that exhibits unique advantages in solving small-sample, nonlinear, and high-dimensional pattern recognition problems. Its effectiveness in small-sample classification has been validated in text classification, handwritten digit recognition, and natural language processing. The principle of SVM is to map the input vector X to a high-dimensional feature space using a pre-selected nonlinear mapping (kernel function). An optimal classification hyperplane is then constructed in this space to separate the two classes of samples without error, maximizing the gap between them. The former ensures minimal empirical risk, while the latter minimizes the confidence range in the generalization bound (i.e., minimizing the structural risk of the binary classifier model). This transforms a nonlinearly separable problem in the original space into a linearly separable problem in a high-dimensional space.

[0073] This embodiment uses the SVM algorithm package in sklearn for actual binary divider model training and prediction. It primarily employs k-fold cross-validation to find the optimal parameters of the SVM-based binary divider model. The parameters requiring optimal parameter selection in this embodiment are C (the penalty term coefficient for relaxation), gamma (the coefficient of the kernel function), and kernel (the choice of kernel function). The steps for determining the optimal parameters using k-fold cross-validation are as follows:

[0074] The first step is to read the sample data, first determine the need to carry out the optimal parameter selection of the parameters C (relaxation coefficient of the penalty term coefficient), gamma (kernel function coefficient), kernel (kernel function selection), and then according to the value range of C, gamma, kernel, randomly generate a set of {C, gamma, kernel} as the initial position of the particle. Among them:

[0075] The value range of C is 0 to 1, and the interval of each value is increased by 0.05, that is, the value range of C is {0, 0.05, 0.1, 0.15, 0.2, 0.25, …, 1};

[0076] The value range of gamma is {auto, rbf, poly, sigmoid}, where auto means the default value, rbf means the radial basis kernel function, poly means the polynomial kernel function, and sigmoid means the hyperbolic tangent kernel function.

[0077] The value range of kernel is {linear, poly, rbf, sigmoid}, where linear means linear kernel function, poly means polynomial kernel function, rbf means radial basis kernel function, and sigmoid means hyperbolic tangent kernel function.

[0078] The second step is to divide a certain training input vector into k non-overlapping subsets S1, S2, …, Sk.

[0079] The third step is to initialize e = 1.

[0080] The fourth step is to leave a certain subset Se of the training input vector as the test set, and combine the remaining subsets as the training set. Use these training sets to train the binary classifier model.

[0081] The fifth step is to test the binary classifier model using the test input vector to obtain the comprehensive performance evaluation index value f e of the Se subset. Specifically:

[0082] In this embodiment, the accuracy Precision and recall Recall are used as the evaluation indicators of the binary classifier model. The formula of the accuracy Precision is defined as follows: The formula of the recall Recall is defined as follows: Where TP represents the number of correctly classified metadata sensitive types, FP represents the number of metadata sensitive types that are not sensitive but are incorrectly predicted to be sensitive, and FN represents the number of metadata sensitive types that are sensitive but are incorrectly predicted to be not sensitive. After training the binary classifier model using the subset of training input vectors as the training set, the entire test input vector is input into the trained binary classifier model for testing, and the test result is compared with the corresponding sensitive type classification result of the first test set data in the sample data table shown in Table 1. If the test result of a certain test input vector is the same as the sensitive type classification result of the corresponding first test set data, it is marked as correctly classified metadata sensitive types. If the test result of a certain test input vector is sensitive but the sensitive type classification result of the corresponding first test set data is not sensitive, it is marked as metadata sensitive types that are not sensitive but are incorrectly predicted to be sensitive. If the test result of a certain test input vector is not sensitive but the sensitive type classification result of the corresponding first test set data is sensitive, it is marked as metadata sensitive types that are sensitive but are incorrectly predicted to be not sensitive. Then, the number of each mark is counted to obtain the values of TP, FP, and FN. Figure 2

[0083] The higher the values of accuracy Precision and recall Recall, the better the effect of the binary classifier model. However, the recall Recall reflects the ratio of how many actual positive samples are correctly classified. When the recall Recall is high, the accuracy Precision is often low, and vice versa. Therefore, the weighted harmonic mean of the recall Recall and the accuracy Precision is used as the comprehensive performance evaluation index value f e of the binary classifier model. e The value range of the comprehensive performance evaluation index value f e is [0, 1], and the specific formula definition is as follows:

[0084] Step 6: Let e = e + 1.

[0085] Step 7: Repeat steps 4 and 5 until e = k, and record the comprehensive performance evaluation index value f e corresponding to each e.

[0086] Step 8: Calculate the generalization comprehensive evaluation score F of this group {C, gamma, kernel}. The generalization comprehensive evaluation score F is the sum average of the comprehensive performance evaluation index value f e corresponding to each e, and the specific formula is as follows:

[0087] Step 9: Repeat steps 1 to 8 until all combinations of {C, gamma, kernel} are traversed.

[0088] Step 10: Select the set of {C, gamma, kernel} with the maximum generalized comprehensive evaluation score Fmax as the optimal parameters of the binary classifier model, and then reset the binary classifier model based on the SVM support vector machine with the optimal parameters to obtain a trained binary classifier model.

[0089] Then the system uses the trained binary classifier model to divide the sensitive types of the remaining specific information of the field metadata, and if the division result includes sensitive and insensitive, the binary classifier model can automatically divide the remaining specific information into sensitive or insensitive.

[0090] A5. Obtain the sensitive types of all specific information of the field metadata respectively, and count the proportion of specific information with sensitive type as sensitive. If the proportion is not less than the preset threshold, the field metadata is divided into sensitive field, and if the proportion is less than the preset threshold, the field metadata is divided into insensitive field.

[0091] Taking the field metadata "name" as an example, after using the trained binary classifier model to divide the remaining specific information of the field metadata "name" into sensitive or insensitive, among all the specific information (500 items) of the field metadata "name", there are part of the specific information (50 items) with sensitive types divided by business personnel and the remaining specific information (450 items) with sensitive types divided by the binary classifier model. Then the system can obtain the sensitive types of all specific information of the field metadata "name" respectively, and count the proportion of specific information with sensitive type as sensitive. If the proportion is not less than the preset threshold (specifically 90%), the field metadata is divided into sensitive field, and if the proportion is less than the preset threshold, the field metadata is divided into insensitive field. For example, if among all the specific information of the field metadata "name", 490 items of specific information are divided into sensitive and 10 items of specific information are divided into insensitive, the proportion of specific information with sensitive type as sensitive is 490 / 500*100% = 98%, which is not less than the preset threshold, so the system divides the field metadata "name" into sensitive field. For another example, if among all the specific information of the field metadata "name", 400 items of specific information are divided into sensitive and 100 items of specific information are divided into insensitive, the proportion of specific information with sensitive type as sensitive is 400 / 500*100% = 80%, which is less than the preset threshold, so the system divides the field metadata "name" into insensitive field.

[0092] B. Divide the topics of table metadata, specifically including the following steps B1-B5.

[0093] B1. Obtain multiple table metadata from the data warehouse, and obtain the results of the division of the topics of part of the table metadata by business personnel according to business application classification.

[0094] In the subject division of table metadata, the system first obtains multiple table metadata stored in the data warehouse, and then the business personnel can divide the subject of part of the table metadata according to the business application classification. For example, there are 300 table metadata in the data warehouse, such as: the first table metadata, the table name is table_1, the table description is "this table stores the relevant information of user buying stocks", the table fields are "user name, purchase date, stock code ……"; the second table metadata, the table name is table_2, the table description is "this table stores the relevant information of user buying insurance", the table fields are "user name, purchase date, insurance code ……"; the third table metadata …… Then, the business personnel can divide the subject of 30 table metadata according to the business application classification, and the remaining 270 table metadata are not divided into subjects, for example, the first table metadata is divided into the theme of finance according to the table description and table field, the second table metadata is divided into the theme of economic operation, ……

[0095] B2. Divide the part of the table metadata with divided subjects into the second training set data and the second test set data according to the preset proportion, then perform text preprocessing operation on each second training set data, the text preprocessing operation includes merging string, word segmentation operation and removing stop word operation, then input each second training set data and each second test set data into the word vector model based on the word2vec corpus for training, and obtain the trained word vector model.

[0096] After the system obtains the result of the business personnel dividing the subject of part of the table metadata, according to the table name, table description, table field and theme of the part of the table metadata with divided subjects, a sample data table of table metadata as shown in Figure 5 is constructed, and then the system uses the non-replacement random sampling method to randomly select 70% sample data from the sample data table as shown in Figure 5 as the second training set data, and the remaining 30% sample data as the second test set data. Then the system performs text preprocessing operation on each second training set data and each second test set data, and the text preprocessing operation process includes the following three operations:

[0097] (1) Merge the texts corresponding to the same second training set data in the sample data table into a string, and merge the texts corresponding to the same second test set data into a string. For example, the first table metadata is one of the second training set data, the table name is table_1, the table description is "This table stores information about user stock purchases", and the table fields include "user name, purchase date, stock code, and theme divided by business personnel as finance". The text corresponding to the first table metadata is merged into the string "table_1 This table stores information about user stock purchases user name, purchase date, stock code, and theme divided by business personnel as finance".

[0098] (2) Jieba segmentation operation. Jieba is an excellent Chinese word segmentation third-party library that uses a Chinese word library to determine the correlation probability between Chinese characters. Chinese characters with high correlation probability form word groups to form the segmentation result. There are three modes for jieba segmentation operation: the first is the accurate mode, which accurately divides the text without redundant words; the second is the full mode, which scans all possible words in the text with redundancy (i.e., a text may be divided into different words from different angles); and the third is the search engine mode, which further divides long words based on the accurate mode. The jieba segmentation operation in this embodiment uses the accurate mode.

[0099] After merging the string "table_1 This table stores information about user stock purchases user name, purchase date, stock code, and theme divided by business personnel as finance", jieba segmentation operation is performed to obtain multiple words, including "table_1", "this table", "stores", "user", "purchase", "stock", "related information", "user name", "purchase date", "stock code", "finance", and so on.

[0100] (3) Remove stop words operation. Stop words are functional words in human language that have no actual meaning and can save storage space and improve search efficiency. In this paper, the Harbin Institute of Technology stop word list is used to remove stop words from the first training set data.

[0101] After the jieba segmentation operation is performed on the string "table_1 This table stores relevant information of users buying stocks User name, date of purchase, stock code Finance", the system performs a stop word removal operation on the plurality of words, finds that the word "stored" contains the stop word "ed", and the word "of" is a stop word itself, and therefore removes "ed" and "of", obtaining the string segmentation result as "table 1", "this table", "store", "user", "buy", "stock", "relevant information", "user name", "purchase date", "purchase stock code", "finance", ….

[0102] Then the system inputs the strings of each second training set data and each second test set data into the word vector model based on the word2vec corpus for training to obtain a trained word vector model. The trained word vector model maps each word to a vector and can be used to represent the relationship between words.

[0103] B3. Obtain a plurality of preset subject domains, and obtain representative table metadata selected by business personnel from the data warehouse for each subject domain according to business experience, then calculate the feature weight of each word in each representative table metadata, select a plurality of words with a feature weight ranking in a preset position as central words that need to be vectorized in the subject domain to which the representative table metadata belongs, and then input each central word into the trained word vector model to obtain a vectorized representation of each central word, thereby obtaining a plurality of central word vectors.

[0104] After obtaining the trained word vector model, the system obtains a plurality of preset subject domains according to the opinions of business personnel, including the following eight: finance, economic operation, macro economy, news information, double carbon monitoring, business environment, background management, and terminal management. Moreover, the system obtains representative table metadata selected by business personnel from the data warehouse for the above eight subject domains according to business experience, that is, selects one most representative table metadata for each subject domain, and then performs a merging string, segmentation, and stop word removal operation on these representative table metadata. Then the system uses the same feature weight calculation method as in step A4 above, calculates the feature weight of each word in the representative table metadata using the TF-IDF method, and then sorts each word according to the feature weight from large to small, selects the top 20 words for the corresponding subject domain as the central words that need to be vectorized in the subject domain, and records them as v j , where j takes a value from 1 to 20, and the central word set under each subject domain is MV i = (v1, v2, …, v 20 ), where v1~v 20 represents each central word, and MV ii represents the number of items in one of the eight subject domains, and takes values from 1 to 8, and the subject domains represented by 1 to 8 in turn are finance, economic operation, macro economy, news information, double carbon monitoring, business environment, background management, and terminal management.

[0105] Then, the system inputs each set of central words into the trained word vector model, obtains the vectorized representation of each central word using the word vector model, and obtains a plurality of central word vectors Where i represents one of the eight subject domains, and takes values from 1 to 8, j takes values from 1 to 20, and kj represents the result of vectorization of the central word with the top 20 feature weight sizes, and the set of central word vectors under each subject domain is Where represents each central word vector.

[0106] B4. Perform text preprocessing operations on the remaining table element data, including merging strings, tokenization, and removing stop words, to obtain the string tokenization result of the remaining table element data. The detailed process is not described again.

[0107] The system performs the same text preprocessing operations on the remaining 270 table element data as in step B2 above, which also includes merging strings, tokenization, and removing stop words, to obtain the string tokenization result of the remaining table element data.

[0108] B5. For each remaining table element data, calculate the cosine similarity between the string tokenization result and each central word vector, obtain the maximum cosine similarity, and then obtain the subject domain to which the central word vector corresponding to the maximum cosine similarity belongs. The remaining table element data is divided into this subject domain.

[0109] Then, the system calculates the cosine similarity between the string tokenization result and each central word vector for each remaining table element data, and obtains the maximum cosine similarity. For example, in step B4, the string tokenization result obtained after preprocessing operation of a certain remaining table element data is denoted as Where N represents the number of items of the string (remaining table element data) to be divided into a subject domain, and takes values from 1 to 270, and m represents the total number of string tokenization, so the set of string tokenization results of the remaining table element data can be denoted as Then, the same feature weight calculation method as in step B3 above is used to calculate the feature weight of the string tokenization result of each remaining table element data using the TF-IDF method, and each tokenization result is sorted according to the feature weight from large to small. The top 20 words are selected as the feature word set of the string tokenization result, denoted as Then the feature word set The input is fed into a trained word vector model, which is then used to obtain the vectorized representation of each feature word, resulting in 20 feature word vectors. Where N represents the number of string (remaining table metadata) items to be partitioned into subject domains, then Let represent the first feature word vector of the Nth remaining table metadata. This represents the second feature word vector of the Nth remaining table data, thus the set of feature word vectors can be obtained as follows: Then, for each remaining table metadata, the system uses the cosine similarity algorithm to calculate the feature word vector set vt of that remaining table metadata. N With the central word vector set VV under each topic domain i The cosine similarity is calculated using the following formula:

[0110]

[0111] Since there are 8 topic domains, namely the central word vector set VV i The value of i in the set ranges from 1 to 8, therefore, for each remaining table metadata feature word vector set vt N All of them can calculate 8 sets of vectors VV corresponding to different center words. i The system calculates the cosine similarity of the eight values ​​and then selects the maximum value from them, using the following formula:

[0112] maxcosVV i vt N =max{cosVV1vt N ,cosVV2vt N ,…,cosVV8vt N};

[0113] Then, the system obtains the corresponding center word vector VV based on the selected maximum cosine similarity. i The subject domain to which the remaining table data belongs will be assigned to that subject domain.

[0114] For example, for the first remaining table metadata, its feature word vector set vt N If N is 1, then Calculate the feature word vector set vt1 of the first table metadata and the center word vector set VV under each topic domain. i The formula for cosine similarity is:

[0115]

[0116] The formula for calculating the maximum cosine similarity is: maxcosVV ivt1 = max{cosVV1vt1, cosVV2vt1, …, cosVV8vt1}; if the value of cosVV2vt1 is the maximum, it means that the characteristic word vector vt1 of the first table metadata has the maximum cosine similarity with the central word vector set VV2 under the second subject domain "economic operation", that is, the first table metadata belongs to the second subject domain "economic operation", so the remaining table metadata is divided into the second subject domain "economic operation".

[0117] In summary, when classifying metadata, the business personnel only need to divide part of the field metadata into sensitive types and part of the table metadata into subjects, and then the system can automatically divide the remaining field metadata into sensitive types based on the trained binary classifier model and automatically divide the remaining table metadata into subjects based on the trained word vector model. This is more time-saving and labor-saving compared with the traditional metadata classification method of manually classifying and entering metadata by business personnel throughout the process.

[0118] The above is only an embodiment of the present application, which does not limit the scope of patent protection. Those skilled in the art can make non-substantial changes or substitutions based on the present application, which still falls within the scope of patent protection.

Claims

1. A metadata classification method based on automatic recognition technology, characterized in that, The method comprises the following steps: A. dividing the sensitive type of field metadata, specifically comprising the following steps A1-A5; A1. selecting the field metadata requiring sensitive type division from the plurality of table metadata stored in the data warehouse, and finding the associated plurality of specific information from the plurality of table metadata stored in the data warehouse according to the field metadata; A2. obtaining the result of the sensitive type division of part of the specific information of the field metadata by the business personnel according to business experience and / or business rules, which includes sensitive and insensitive; A3. dividing the part of the specific information of the field metadata which has been divided into the sensitive type into a first training set data and a first test set data according to a preset proportion, and then performing a text preprocessing operation on each first training set data, the text preprocessing operation comprising a word segmentation operation and a stop word removal operation; A4. calculating the feature weight of each first training set data and each first test set data, generating a training input vector according to the feature weight of each first training set data, and generating a test input vector according to the feature weight of each first test set data, then inputting each training input vector and each test input vector into a binary classifier model based on a support vector machine (SVM) for training, obtaining a trained binary classifier model, and then using the trained binary classifier model to divide the remaining specific information of the field metadata into the sensitive type, the division result including sensitive and insensitive; A5. obtaining the sensitive type of each of the specific information of the field metadata, calculating the proportion of the specific information with the sensitive type, if the proportion is not less than a preset threshold, dividing the field metadata into a sensitive field, and if the proportion is less than the preset threshold, dividing the field metadata into an insensitive field; B. dividing the theme of table metadata, specifically comprising the following steps B1-B5; B1. obtaining a plurality of table metadata from the data warehouse, and obtaining the result of the theme division of part of the table metadata by the business personnel according to business application classification; B2. dividing the part of the table metadata which has been divided into the theme into a second training set data and a second test set data according to a preset proportion, then performing a text preprocessing operation on each second training set data, the text preprocessing operation comprising a string merging operation, a word segmentation operation and a stop word removal operation, and then inputting each second training set data and each second test set data into a word vector model based on a word2vec corpus for training, obtaining a trained word vector model; B3. obtaining a plurality of preset theme domains, and obtaining the representative table metadata selected by the business personnel from the data warehouse for each theme domain according to business experience, then calculating the feature weight of each word in each representative table metadata, selecting a plurality of words with a feature weight size in a preset rank as the center words which need to be vectorized in the theme domain to which the representative table metadata belongs, and then inputting each center word into the trained word vector model to obtain the vectorization representation of each center word, obtaining a plurality of center word vectors; B4. performing text preprocessing operations on the remaining table metadata, the text preprocessing operations including a string merging operation, a word segmentation operation, and a stop word removing operation, to obtain a word segmentation result of the strings of the remaining table metadata; B5. for each of the remaining table metadata, calculating cosine similarity between the word segmentation result of the strings of the remaining table metadata and each of the central word vectors, obtaining a maximum cosine similarity, and then obtaining a subject domain to which a central word vector corresponding to the maximum cosine similarity belongs, and dividing the remaining table metadata into the subject domain.

2. The method of claim 1, wherein the metadata is classified based on automatic recognition technology. In the step A4, the feature weight is specifically calculated as follows: obtaining a plurality of words obtained after the word segmentation operation, and calculating a TFIDF value of each of the words, specifically as follows: TFIDF = TF × log(n / (m+1)); wherein, TF represents a term frequency of a word, n represents a total number of specific information under the field metadata, and m represents a number of specific information under the segment metadata containing the word.

3. The automatic identification technology based metadata classification method of claim 1, wherein, In the step A4, the SVM-based binary classifier model is trained specifically as follows: an optimal parameter of the SVM-based binary classifier model is found out by using a k-fold cross-validation method, and then the SVM-based binary classifier model is re-set with the optimal parameter to obtain the trained binary classifier model.

4. The method of claim 3, wherein the metadata is classified based on the automatic recognition technology. In the step A4, the optimal parameter of the SVM-based binary classifier model is found out by using the k-fold cross-validation method specifically as follows: First step: determining parameters to be selected for optimal parameters as a penalty term coefficient C of a relaxation coefficient, a coefficient gamma of a kernel function, and a selection kernel of the kernel function, and then randomly generating a set {C, gamma, kernel} as an initial position of a particle according to a value range of the parameters; Second step: dividing a training input vector into k mutually exclusive subsets S1, S2, …, Sk; Third step: initializing e = 1; Fourth step: leaving a subset Se of the training input vector as a test set, and combining the remaining subsets as a training set, and training the binary classifier model with the training set; Step 5: Test the binary classifier model with the test input vector to obtain the comprehensive performance evaluation index value f of the Se-th subset e ; Sixth step: e = e + 1; Step 7: Repeat the fourth and fifth steps until e=k, and record the value of the comprehensive performance evaluation index f corresponding to each e e ; Step 8: Calculate the general comprehensive evaluation score F of this group of {C, gamma, kernel}, which is the general performance evaluation index value f corresponding to each e, which is the average of the sum of the values of the performance evaluation index of each e, and the specific formula is as follows: e ​ Ninth step: repeating the first step to the eighth step until all value combinations of {C, gamma, kernel} are traversed; Tenth step: selecting a set {C, gamma, kernel} with a maximum generalization comprehensive evaluation score F as the optimal parameter of the binary classifier model.

5. The automatic identification technology based metadata classification method of claim 4, wherein, The comprehensive performance evaluation index value f e The formula is defined as: wherein Precision is the accuracy rate, and its formula is defined as Recall is the recall rate, and its formula is defined as wherein TP represents the number of correctly classified metadata sensitive types, FP represents the number of metadata sensitive types that are not sensitive but are incorrectly predicted as sensitive, and FN represents the number of metadata sensitive types that are sensitive but are incorrectly predicted as not sensitive.

6. The automatic identification technology based metadata classification method of claim 1, wherein, In the step B3, the plurality of words ranked in the top 20 in terms of the feature weight size are selected as the central words to be vectorized under the subject domain to which the representative table metadata belongs.

7. The automatic identification technology based metadata classification method of claim 1, wherein, In the step B5, the cosine similarity calculation formula is as follows: wherein, V i represents the set of central word vectors under each topic domain, i represents the number of items in one of the multiple topic domains; vt N represents the set of feature word vectors of the remaining table metadata, N represents the number of items of the remaining table metadata to be divided into topic domains.

8. The automatic identification technology based metadata classification method of claim 1, wherein, In the step A3 and the step B2, the preset ratio is 7:

3.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps in the metadata classification method according to any one of claims 1 to 8.

10. A metadata classification system based on automatic recognition technology, comprising a computer readable storage medium and a processor connected to each other, characterized in that, The computer readable storage medium is as claimed in claim 9.

Images