A data governance management system

Abstract

The present invention belongs to the technical field of data management, and discloses a data governance management system, which includes: collecting original company data from n data sources within a company, and preprocessing the original company data to obtain a standard company data set; analyzing the standard company data set to obtain structured data features and unstructured data information entropy; performing m-round encryption on the original company data based on the structured data features and unstructured data information entropy to generate encrypted data; storing the encrypted data, and intelligently managing the encrypted data. It provides a comprehensive, intelligent, and secure data governance solution for the company, which can significantly enhance the company's data asset value.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and more specifically, to a data governance management system. Background Art

[0002] The patent application publication number CN113094416A discloses a company asset data management system, including a firewall module, a central control module, a login module, an access module, a database module, a data processing module, a data output module and a user terminal; when in use, it can satisfy the user's upload and download of company asset data through the access module, and the database module can classify and store the company asset data; the data processing module reads and processes the company asset data uploaded by the user through the access module, and the data output module can generate financial statements based on the asset data processed by the data processing module, and back up and upload the financial statements for publication; it can not only quickly and accurately classify and integrate the uploaded asset data, but also automatically generate reports for publication, greatly saving the workload of asset data maintenance and management.

[0003] However, existing data governance methods of companies still face many challenges, which seriously restrict the ability of companies to fully utilize data resources. First, the integration efficiency of multi-source heterogeneous data within the company is low. For example, it is difficult for multinational companies to unify ERP system data in different regions. Second, data value mining is insufficient, especially in combining structured and unstructured data, which causes companies to miss important market insights. Data security mechanisms are generally rigid and encryption strategies cannot be adjusted dynamically, increasing the risk of data leakage. At the same time, traditional encryption methods often reduce data utilization efficiency and affect the timeliness of data analysis. Although many companies have a large amount of data, they lack effective tools to convert it into decision support, resulting in excessive reliance on experience and intuition in decision-making. In addition, companies generally lack methods to evaluate and enhance the value of data assets, resulting in insufficient use of data resources. Finally, existing data governance methods of companies can usually only solve some data governance problems, lack comprehensive, intelligent and secure integrated solutions, and increase management complexity and costs. Summary of the invention

[0004] In view of this, the present invention proposes a data governance management system, which provides a company with a comprehensive, intelligent and secure data governance solution, which can significantly enhance the company's data asset value.

[0005] A data governance management system, comprising:

[0006] The data collection module is used to collect the original company data from n data sources within the company and pre-process the original company data to obtain a standard company data set;

[0007] Data analysis module, used to analyze standard company data sets to obtain structured data features and unstructured data information entropy;

[0008] The encryption module performs m rounds of encryption on the original company data based on the structured data characteristics and the information entropy of the unstructured data to generate encrypted data;

[0009] The data storage management module is used to store encrypted data and intelligently manage the encrypted data. The modules are connected via wired and / or wireless means.

[0010] Furthermore, the original company data includes business data, company core data and log data;

[0011] The business data includes customer information forms, order information forms, product information forms and customer feedback; the company's internal data includes bidding data, employee information forms and internal company documents.

[0012] Furthermore, the frequent patterns of original company data from different data sources are obtained, and the original company data are integrated based on the frequent patterns to obtain a standard company data set;

[0013] The frequent pattern acquisition method includes:

[0014] Clean the original company data and determine the transaction object for each business entity based on the structure of the original company data. If it is row-level data, each row record is regarded as a transaction object. If it is document-level data, each document is regarded as a transaction object.

[0015] Select all attributes of the transaction object as item sets; all attributes include categorical attributes, numerical attributes, and set attributes;

[0016] Construct a transaction data set, traverse each transaction object, and for categorical attributes, use the attribute value as the item of the corresponding transaction object in the transaction data set; for numerical attributes, perform discretization to obtain the corresponding discrete value, and map the discrete value to N interval items corresponding to the transaction object in the transaction data set; for set attributes, use each element in the set as the item of the corresponding transaction object in the transaction data set;

[0017] Calculate transaction objects based on transaction data sets Each item Support ;

[0018] ;

[0019] in, For transaction objects The comprehensive weight of For item and item The strength of the association between

[0020] ;

[0021] in For transaction objects The number of middle terms;

[0022] ;

[0023] in, For item and item The frequency of simultaneous occurrence, For item and item The semantic similarity score of For item and item In the transaction object The average distance of and is the weight hyperparameter; item The frequency of occurrence, For item The frequency of occurrence, To obtain the maximum value;

[0024] Preset the minimum support threshold min_sup, all Items greater than min_sup Arrange in descending order according to the size of support to generate a frequent item header table; build an FP tree based on the frequent item header table to obtain frequent patterns.

[0025] Furthermore, the method of constructing the FP tree based on the frequent item header table to obtain the frequent pattern includes:

[0026] Create a root node as an empty FP tree. For each transaction object, insert the frequent item header table as a path into the FP tree. Starting from the root node, for each node on the path, if there is a child node that is the same as the current item, increase the child node count. Otherwise, create a new node as a child node and link it to the path of the corresponding node in the frequent item header table. The count of the last node in the path is set to 1.

[0027] For each node p, traverse the corresponding linked list to obtain the support count L_p of node p;

[0028] If L_p is greater than the preset counting threshold Tup, then node p forms a frequent pattern of length 1;

[0029] For each frequent pattern P whose length is greater than 1, traverse all paths associated with P on the FP tree, form the paths prefixed by P into a conditional pattern base, build a conditional FP tree of P based on the conditional pattern base, and recursively mine on the conditional FP tree of P to obtain frequent patterns prefixed by P.

[0030] Further, the method of constructing the conditional FP tree of P according to the conditional pattern base includes:

[0031] For a frequent pattern P, traverse all paths associated with P on the FP tree. Each path is a suffix pattern, denoted as a. All suffix patterns a form a conditional pattern base.

[0032] Calculate the frequency freq(a) of each suffix pattern a in the conditional pattern base;

[0033] Filter out the suffix patterns whose frequency freq(a) is less than the preset frequency threshold, and take the remaining suffix patterns as frequent suffix patterns b;

[0034] Calculate the support count Xb of b, build the path of the conditional FP tree corresponding to b based on Xb, combine the paths of all conditional FP trees, and build the conditional FP tree of P.

[0035] Furthermore, the method of integrating the original company data includes:

[0036] Define a pattern tree. Each frequent pattern corresponds to a pattern tree. The root node of the pattern tree is empty. Each node corresponds to an item in the frequent pattern. The connection relationship between nodes represents the order of items in the pattern.

[0037] Calculate the structural similarity between different pattern trees. The calculation formula for structural similarity is:

[0038] ;

[0039] in, for and The number of nodes in the largest common subtree of For the model tree The number of nodes, For the model tree The number of nodes; For the model tree and the schema tree The structural similarity between

[0040] The similarity between nodes is calculated by the node similarity function, and the weighted sum of the structural similarity and the node similarity is taken as the overall similarity between the pattern trees;

[0041] A similarity threshold is preset, and all pairs of pattern trees whose overall similarity is greater than the similarity threshold are regarded as high-similarity pattern pairs;

[0042] Identify the node pairs on the same level on the high-similarity pattern pairs. For the node pairs on the same level, if they have the same number of child nodes and the relative position order of the child nodes, merge the corresponding node pairs into one node. During the merging process, retain the parallel relationship of the nodes.

[0043] For high-similarity pattern pairs, if the entity types are the same or have semantic associations, they are fused into one node; the set of pattern trees obtained after the fusion of all high-similarity pattern pairs is used as the unified pattern representation of the original company data, that is, the integration of the original company data is completed to obtain a standard company data set.

[0044] Furthermore, the and The methods for obtaining the largest common subtree include:

[0045] Will and Represented as two sets and ,in express A node in express A node in

[0046] Construct a two-dimensional matrix M1. The element M1[x][y] in the two-dimensional matrix M1 represents and is the size of the largest common subtree of the two subtrees of the root node; where x is the row index of the two-dimensional matrix M1, and y is the column index of the two-dimensional matrix M1;

[0047] For any element M1[x][y] in the two-dimensional matrix M1, if The uth node and The vth node The signs are the same, then M1[x][y]=M1[x-1][y-1]+1;

[0048] like and If the signs are different, then M1[x][y]=max(M1[x-1][j], M1[x][y-1]);

[0049] Then the last element M1[m2][n2] of the two-dimensional matrix M1 is and The size of the largest common subtree of the two trees; m2 is the last row of the two-dimensional matrix M1, and n2 is the last column of the two-dimensional matrix M1; by backtracing the two-dimensional matrix M1, the structure of the largest common subtree is reconstructed; that is, the acquisition of the largest common subtree is completed.

[0050] Furthermore, the method of acquiring the structured data features includes:

[0051] Segment the structured data in the standard company data set to obtain F1 words, and calculate the local occurrence probability, word distribution density value, and word association information value matrix of each word; the local occurrence probability, word distribution density value, and word association information value matrix of each word are the structured data features;

[0052] Count the attribute values ​​of each word Probability of occurrence ; then the local occurrence probability of a word ;

[0053] in, is the total number of data entries, For attribute value The global weight of

[0054] ;

[0055] in, For attribute value The value of distinguishes the ability weights, For attribute value The semantic weight of the value; is the balance coefficient;

[0056] ;

[0057] in, For attribute value The value range of values, For attribute value The value range of The probability of occurrence of a value; For attribute value The total number of values ​​in the range of ;

[0058] ;

[0059] in, for The semantic importance score of for The semantic importance score of For attribute value The value range of values;

[0060] For each word , its word distribution density value ;

[0061] in, For words The frequency of the document in the entire text collection, For words The complement of the document frequency in the entire text collection;

[0062] For any two words and , calculate the point mutual information value between the two ;

[0063] ;

[0064] in, For words and The probability of simultaneous occurrence, For words The probability of a single occurrence, For words Probability of single occurrence;

[0065] Initially, an empty matrix is ​​created, and the words are used as the rows and columns of the empty matrix. The corresponding point mutual information values ​​are used as the corresponding matrix elements, and the word association information value matrix is ​​obtained.

[0066] Furthermore, the method for obtaining the information entropy of the unstructured data includes:

[0067] Convert the unstructured data in the standard company data set into text form to obtain text data; remove stop words from the text data to obtain perfect text data;

[0068] The perfect text data is converted into a set of points in a high-dimensional vector space, and a regular grid is constructed to divide the high-dimensional vector space into Cube;

[0069] Compute the minimum number of cubes needed to cover a set of points ; Then we get the box counting dimension The calculation formula is:

[0070] ;

[0071] In scale Under the condition of The probability of a point contained in ;

[0072] And calculate and improve the text data at scale Information entropy under ;

[0073] Based on information entropy , calculate the information dimension ;

[0074] Get the matrix of distances between points, for a given scale , the statistical distance is less than The number of point pairs ; Based on point logarithm , calculate the correlation dimension ;

[0075] The information entropy of unstructured data is ;

[0076] in, , and is the corresponding weight coefficient; is the index parameter.

[0077] Furthermore, the original company data is Round encryption methods include:

[0078] Defines the number of rounds to encrypt raw corporate data ;

[0079] in, is the data sensitivity score, score points for safety requirements; is the sensitivity weight, Weight for safety requirements;

[0080] Data Sensitivity Score ;

[0081] in, Score for confidentiality, For the value score, is the score balance weight parameter;

[0082] ;

[0083] in, To score the risk assessment, Score your company's compliance, is the risk weight, is the compliance weight;

[0084] Generate a first key part based on the structured data feature, and generate a second key part based on the unstructured data information entropy;

[0085] The first key part is generated by:

[0086] Sort the local occurrence probabilities of all words, take the values ​​of the first N3 local occurrence probabilities, and concatenate them into a string S1; Sort the word distribution density values ​​of all words, take the first M3 values, and concatenate them into a string S2; Extract the non-zero elements in the diagonal elements, upper triangular elements, or lower triangular elements of the word association information value matrix, and concatenate them into a string S3; Concatenate S1, S2, and S3 into a string S4, and perform a hash operation on S4 to obtain a fixed-length hash value H1, i.e., the first key part;

[0087] The first key part is generated in the following manner:

[0088] Take the decimal part of the information entropy of the unstructured data to obtain a decimal X1 between 0 and 1, multiply X1 by a preset integer M5, and take the remainder of M5 to obtain an integer Y1; convert Y1 into a fixed-length string S5, perform a hash operation on S5, and obtain a fixed-length hash value H2, i.e., the second key part;

[0089] Concatenate the first key part and the second key part to generate an m-round encryption key;

[0090] Using the encryption key, m rounds of encryption are performed on the original company data. In each round, different encryption algorithms and encryption keys are used to encrypt the encrypted data obtained in the previous round to obtain new encrypted data, until the mth round, the final encrypted data is obtained.

[0091] Technical effects and advantages of a data governance management system of the present invention:

[0092] The present invention adopts frequent pattern mining and pattern tree fusion to efficiently integrate multi-source heterogeneous data within the company and realize data processing. It combines the comprehensive analysis of structured data characteristics and unstructured data information entropy to deeply explore the value of data and provide strong support for company decision-making. The system adopts a dynamic encryption mechanism based on data sensitivity and security requirements, and generates encryption keys in combination with data characteristics, which greatly improves data security. The intelligent encrypted data storage and management function optimizes the utilization efficiency of data, which helps the company to better realize data-driven, improve operational efficiency, explore new business opportunities, and enhance market competitiveness. In addition, it ensures the compliance of the company's data processing and reduces compliance risks. It provides the company with a comprehensive, intelligent and secure data governance solution, which can significantly enhance the company's data asset value. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 This is a schematic diagram of a data governance management system of the present invention. DETAILED DESCRIPTION

[0094] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0095] Embodiment 1

[0096] See also Figure 1 As shown, a data governance management system described in this embodiment includes:

[0097] The data collection module is used to collect the original company data from n data sources within the company and pre-process the original company data to obtain a standard company data set;

[0098] Data analysis module, used to analyze standard company data sets to obtain structured data features and unstructured data information entropy;

[0099] The encryption module performs m rounds of encryption on the original company data based on the structured data characteristics and the information entropy of the unstructured data to generate encrypted data;

[0100] The data storage management module is used to store encrypted data and intelligently manage the encrypted data. The modules are connected via wired and / or wireless means to achieve data transmission between modules.

[0101] The original company data includes business data, company core data and log data;

[0102] The business data includes customer information table (customer ID, customer name, contact information, contact address), order information table (order number, order time, product list), product information table (product code, product name, price) and customer feedback;

[0103] The company's internal data includes bidding data (project number, bid price and winning bid amount), employee information table (employee number, name and department) and internal company documents (reports and meeting minutes);

[0104] Log data is the operation record automatically recorded by the company's applications during operation; original company data is the key prerequisite for realizing data-driven, which helps companies improve operational efficiency, discover new opportunities, enhance competitiveness, and ensure compliance and data security.

[0105] The method of preprocessing the original company data includes:

[0106] Obtaining frequent patterns of original company data from different data sources and integrating the original company data based on the frequent patterns will make the data structure consistent and obtain a standard company data set.

[0107] The frequent pattern acquisition method includes:

[0108] Clean the original company data, including processing missing values ​​and outliers;

[0109] Convert the original company data into a transaction data set, where each transaction is a set of attribute values ​​of a data object; specifically, based on the structure of the original company data, such as tables and files, identify business entities such as customers, orders, products, etc.

[0110] For each business entity, determine its transaction object. If it is row-level data (referring to structured data, such as table data; each row (or record) represents an independent data entity and contains information of multiple columns (or fields)), then each row record is a transaction object. If it is document-level data (semi-structured or unstructured data, each document is an independent data unit and may contain complex nested structures), then each document is a transaction object.

[0111] Select all attributes of the transaction object as item sets (item sets are the basic element sets that constitute transaction objects); all attributes include categorical attributes (data that can be divided into different categories or groups), numerical attributes (numerical data that can perform mathematical operations), and set attributes (attributes that indicate that an entity can have multiple values ​​at the same time); for example, select the customer's name, age, address, etc. as item sets, and select the order number, order time, product list, etc. of the order as item sets.

[0112] Construct a transaction data set and traverse each transaction object. For categorical attributes, use the attribute value as the item of the corresponding transaction object in the transaction data set. For numerical attributes, perform discretization to obtain the corresponding discrete value and map the discrete value to N interval items corresponding to the transaction object in the transaction data set. For set attributes, use each element in the set as the item of the corresponding transaction object in the transaction data set.

[0113] Transactional datasets are represented as lists or sets, for example, [(item1, item2, item4), (item1, item3), (item2, item3, item4)] or [(item1, item2, item4), (item1, item3), (item2, item3, item4)].

[0114] Each transaction in the transaction data set is a set of items. Different transactions are independent of each other, and items can appear repeatedly in different transactions.

[0115] Calculate transaction objects based on transaction data sets Each item Support ;

[0116] ;

[0117] in, For transaction objects The comprehensive weight of For item and item The strength of the association between

[0118] ;

[0119] in For transaction objects The number of middle items.

[0120] ;

[0121] in, For item and item The frequency of simultaneous occurrence, For item and item The semantic similarity score of (calculated by word vector similarity), with a value range of [0, 1]; For item and item In the transaction object The average distance of (can be based on position or sequence); and is the weight hyperparameter; item The frequency of occurrence, For item The frequency of occurrence, To get the maximum value.

[0122] Preset the minimum support threshold min_sup, all Items greater than min_sup Arrange in descending order according to the size of support to generate a frequent item header table; build an FP tree based on the frequent item header table to obtain frequent patterns.

[0123] Specifically, create a root node as an empty FP tree. For each transaction object, insert the frequent item header table as a path into the FP tree. Starting from the root node, for each node on the path (each node represents an item), if there is a child node that is the same as the current item, increase the child node count. Otherwise, create a new node as a child node and link it to the path of the corresponding node in the frequent item header table. The count of the last node in the path is set to 1.

[0124] For each node p, traverse the corresponding linked list to obtain the support count L_p of node p (the number of transaction objects containing node p); if L_p is greater than the preset count threshold Tup, then node p forms a frequent pattern with a length of 1.

[0125] For each frequent pattern P whose length is greater than 1, traverse all paths associated with P on the FP tree, form the paths prefixed by P into a conditional pattern base, build a conditional FP tree of P based on the conditional pattern base, and recursively mine on the conditional FP tree of P to obtain frequent patterns prefixed by P.

[0126] Specifically, for a frequent pattern P, all paths associated with P on the FP tree are traversed, each path is a suffix pattern, denoted as a, and all suffix patterns a form a conditional pattern base.

[0127] Calculate the frequency freq(a) of each suffix pattern a in the conditional pattern base; filter out the suffix patterns whose frequency freq(a) is less than the preset frequency threshold, and use the remaining suffix patterns as frequent suffix patterns b; specifically, the frequency threshold can be set as a percentage of the total number of records, for example, 1% or 0.5%.

[0128] Calculate the support count Xb of b, and build the path of the conditional FP tree corresponding to b based on Xb. Specifically, for each path in the conditional pattern base, retain the frequent items (items with support greater than the minimum support threshold), use the support count of b as the initial count of these items, and insert these processed paths into the new FP tree structure. If different paths have the same items, merge these nodes and accumulate their counts; combine the paths of all conditional FP trees to build the conditional FP tree of P.

[0129] The methods for integrating the original company data include:

[0130] Define a pattern tree. Each frequent pattern corresponds to a pattern tree. The root node of the pattern tree is empty. Each node corresponds to an item in the frequent pattern. The connection relationship between nodes represents the order of items in the pattern.

[0131] Calculate the structural similarity between different pattern trees. The calculation formula for structural similarity is:

[0132] ;

[0133] in, for and The number of nodes in the largest common subtree of For the model tree The number of nodes, For the model tree The number of nodes; For the model tree and the schema tree The structural similarity between them.

[0134] and The methods for obtaining the largest common subtree include:

[0135] Will and Represented as two sets and ,in express A node in express A node in .

[0136] Construct a two-dimensional matrix M1. The element M1[x][y] in the two-dimensional matrix M1 represents and is the size of the largest common subtree of the two subtrees of the root node; where x is the row index of the two-dimensional matrix M1, and y is the column index of the two-dimensional matrix M1.

[0137] For any element M1[x][y] in the two-dimensional matrix M1, if The uth node and The vth node If the signs are the same, and The largest common subtree of the two subtrees of the root node is and After that, the largest common subtree of the remaining two subtrees plus and For these two nodes, M1[x][y]=M1[x-1][y-1]+1.

[0138] The same label means that two nodes represent the same entity or concept; for example, if one node represents the concept of a person and the other node also represents a person, then their labels are the same.

[0139] like and If the mark is different, and The largest common subtree of the two subtrees with as the root node is The subtree of the root node is The largest common subtree of the remaining part, and The subtree of the root node is The larger of the largest common subtrees of the remaining parts; that is, M1[x][y]=max(M1[x-1][j], M1[x][y-1]).

[0140] Then the last element M1[m2][n2] of the two-dimensional matrix M1 is and The size (number of nodes) of the largest common subtree of the two trees; m2 is the last row of the two-dimensional matrix M1, and n2 is the last column of the two-dimensional matrix M1; by backtracing the two-dimensional matrix M1, the structure of the largest common subtree is reconstructed.

[0141] The similarity between nodes is calculated through a node similarity function (such as semantic similarity), and the weighted sum of structural similarity and node similarity is taken as the overall similarity between pattern trees.

[0142] A similarity threshold is preset, and all paired pattern trees whose overall similarity is greater than the similarity threshold are regarded as high-similarity pattern pairs.

[0143] Node pairs on the same level are identified on high-similarity pattern pairs. For node pairs on the same level, if they have the same number of child nodes and relative position order of child nodes, the corresponding node pairs are merged into one node; during the merging process, the parallel relationship of the nodes is retained.

[0144] For high-similarity pattern pairs, the entity types represented by the nodes are compared. If the entity types are the same or have semantic relevance (such as belonging to the superordinate concept of "person"), they are merged into one node. During the fusion process, the semantic information of the node, such as entity type, attribute meaning, etc., is retained for subsequent data processing and analysis.

[0145] After the fusion of the above steps, a new pattern tree representation is obtained, which integrates the structural and semantic features of the original pattern pairs; this new pattern tree representation has better generalization ability and can cover more data instances.

[0146] The pattern tree set obtained after fusing all high-similarity pattern pairs is used as the unified pattern representation of the original company data, that is, the integration of the original company data is completed to obtain a standard company data set.

[0147] The method of analyzing the standard company data set includes:

[0148] The structured data in the standard company data set is segmented to obtain F1 words. Structured data refers to data with a fixed or predefined pattern, such as table data in a relational database, XML data, etc.

[0149] Calculate the local occurrence probability, word distribution density value and word association information value matrix of each word; the local occurrence probability, word distribution density value and word association information value matrix of each word are structured data features.

[0150] Specifically, count the attribute values ​​of each word Probability of occurrence (Property value the number of occurrences in the standard company data set divided by the total number of attribute values);

[0151] Then the local occurrence probability of a word is ;

[0152] in, is the total number of data entries (referring to the total number of record entries in the entire standard company data set), For attribute value The global weight of .

[0153] ;

[0154] in, For attribute value The value of distinguishes the ability weights, For attribute value The semantic weight of the value; is the balance coefficient, which is used to balance the weights.

[0155] ;

[0156] in, For attribute value The value range of values, For attribute value The value range of The probability of occurrence of a value; For attribute value The total number of values ​​in the value domain; the value domain of an attribute value refers to the range of all possible values ​​of a certain attribute (or field), for example, for the "gender" attribute, the value domain is {male, female}, for the "age" attribute, the value domain is {0, 1, 2, ..., 100}; for the "education level" attribute, the value domain is {primary school, junior high school, high school, university, graduate school}.

[0157] ;

[0158] in, for The semantic importance score of for The semantic importance score of For attribute value The value range of value.

[0159] For each word , its word distribution density value ;

[0160] in, For words The document frequency (number of documents that appear) in the entire text collection (referring to the text attribute value of structured data), For words The complement of the document frequency in the entire text collection.

[0161] It should be explained that text attribute value refers to the text part of the attribute value in structured data. Structured data is usually stored in the form of a table, where each row is a record and each column is an attribute. Attribute values ​​can be different data types such as numbers, dates, and string text. Text attribute value specifically refers to the data that is part of the string text in the attribute value; for example, in a customer information table:

[0162] For the two attributes of customer name and contact address, their values ​​such as Zhang San and No. xx, xxx Road, Beijing, etc. are all text strings, which are called text attribute values; digital attribute values ​​such as customer ID do not belong to the category of text attribute values.

[0163] For any two words and , calculate the point mutual information value between the two ;

[0164] ;

[0165] in, For words and The probability of simultaneous occurrence, For words The probability of a single occurrence, For words Probability of appearing alone.

[0166] Initially, an empty matrix is ​​created, and the words are used as the rows and columns of the empty matrix. The corresponding point mutual information values ​​are used as the corresponding matrix elements, and the word association information value matrix is ​​obtained. The larger the point mutual information value, the stronger the association between the two words, and the higher the association information value.

[0167] The unstructured data refers to data without a fixed or predefined pattern, such as natural language text.

[0168] The methods for obtaining information entropy of unstructured data include:

[0169] The unstructured data in the standard company data set is converted into text form (if it is already in text form, no conversion is performed) to obtain text data; the text data is processed to remove stop words to obtain complete text data.

[0170] The perfect text data is converted into a set of points in a high-dimensional vector space, and a regular grid is constructed to divide the high-dimensional vector space into The cube, also called the scale cube.

[0171] Compute the minimum number of cubes needed to cover a set of points ; Then we get the box counting dimension The calculation formula is:

[0172] .

[0173] In scale Under the condition of The probability of a point contained in ;

[0174] And calculate and improve the text data at scale Information entropy under ;

[0175] Based on information entropy , calculate the information dimension .

[0176] Get the matrix of distances between points, for a given scale , the statistical distance is less than The number of point pairs ; Based on point logarithm , calculate the correlation dimension .

[0177] The information entropy of unstructured data is ;

[0178] in, , and is the corresponding weight coefficient; is the exponential parameter, which is a positive real number.

[0179] The greater the information entropy of unstructured data, the more dramatic the changes in the statistical characteristics of the data at different scales, the worse the self-similarity, and the higher the degree of disorder and uncertainty.

[0180] Among them, the original company data is Round encryption methods include:

[0181] Defines the number of rounds to encrypt raw corporate data ;

[0182] in, is the data sensitivity score, score points for safety requirements; is the sensitivity weight, Weight for security requirements.

[0183] Data Sensitivity Score ;

[0184] in, Score for confidentiality, For the value score, The score balance weight parameter is used to balance the weights between scores;

[0185] ;

[0186] in, To score the risk assessment, Score your company's compliance, is the risk weight, is the compliance weight.

[0187] It should be noted that the confidentiality score is used to measure the confidentiality of data. Specifically, the main factors affecting confidentiality are determined, such as data type, purpose, difficulty of acquisition, timeliness, legal and regulatory requirements, etc. Scoring standards are designed for each factor. For example, data type scoring: 100 points for bidding data, 90 points for technical data, 80 points for employee data, etc.; purpose scoring: 100 points for core business, 90 points for important auxiliary, 70 points for general purpose, etc.; difficulty of acquisition scoring: 100 points for extremely difficult to obtain, 80 points for relatively difficult to obtain, 60 points for general, and 70 points for relatively easy to obtain. 40 points, 20 points for easy acquisition; Timeliness score: 100 points for current data, 90 points within 1 year, 60 points within 2 years, 30 points within 3 years, 10 points for more than 3 years; Law and regulation score: 100 points for strict compliance, 80 points for certain requirements, 50 points for no special requirements; Set weights for each factor, score each data record according to its attributes corresponding to the scoring criteria, calculate the sum of the weighted scores, and get the confidentiality score of the record; Normalize the confidentiality scores of all records to get a confidentiality score in the interval [0, 1].

[0188] The value score is used to measure the economic value or commercial value of data. Specifically, the main factors affecting the value of data are determined, such as the importance of data to the business, the impact on revenue, the impact on cost, etc., and scoring standards are designed for each factor; for example, business importance score: core business 100 points, important business 80 points, general business 50 points, non-critical business 20 points; revenue impact score: high impact 100 points, large impact 80 points, certain impact 50 points, almost no impact 20 points; cost impact score: high impact 100 points, large impact 80 points, certain impact 50 points, almost no impact 20 points; set weights for each factor, such as business importance weight 0.4, revenue impact weight 0.3, cost impact weight 0.3; for each data record, score according to its attribute corresponding to the scoring standard, calculate the sum of the weighted scores, and normalize them to obtain the value score of the record.

[0189] The risk assessment score is used to measure the degree of data security requirements based on risk assessment. Specifically, it identifies the main risk factors that may lead to data leakage or loss, such as human operational errors, system failures, network attacks, natural disasters, etc.; for each major risk factor, evaluate its probability of occurrence and potential impact, and calculate the risk value, set a weight for each risk factor to reflect its impact on the overall risk, calculate the sum of the weighted risk values, and obtain the overall risk score; map the overall risk score R to a score range, such as [0, 100], to obtain the risk assessment score.

[0190] The company compliance score is used to measure the degree of data security requirements of the company's internal compliance policies. Specifically, collect the company's internal compliance policy documents on data security; extract key compliance requirements from the compliance policy documents, such as encryption strength requirements, access control requirements, audit requirements, data backup requirements, etc.; design scoring standards for each compliance requirement; for example, encryption strength: 60 points for 128-bit encryption and 90 points for 256-bit encryption; access control: 70 points for role-based permission control and 90 points for two-factor authentication; audit requirements: 60 points for log audit and 90 points for real-time audit; data backup: 80 points for off-site backup and 100 points for hot backup. Evaluate the score of each compliance requirement and calculate the sum of the weighted scores to obtain the company's compliance score.

[0191] A first key part is generated based on the structured data feature, and a second key part is generated based on the unstructured data information entropy.

[0192] The first key part is generated by:

[0193] Sort the local occurrence probabilities of all words, take the values ​​of the first N3 local occurrence probabilities, and concatenate them into a string S1; sort the word distribution density values ​​of all words, take the first M3 values, and concatenate them into a string S2; extract the non-zero elements in the diagonal elements, upper triangular elements or lower triangular elements of the word association information value matrix, and concatenate them into a string S3; concatenate S1, S2 and S3 into a string S4, and perform a hash operation on S4 to obtain a fixed-length hash value H1, which is the first key part.

[0194] The first key part is generated in the following manner:

[0195] Take the decimal part of the information entropy of the unstructured data to obtain a decimal X1 between 0 and 1, multiply X1 by a preset integer M5, and take the remainder of M5 to obtain an integer Y1; convert Y1 into a fixed-length string S5, perform a hash operation on S5, and obtain a fixed-length hash value H2, which is the second key part.

[0196] Concatenate the first key part and the second key part to generate an m-round encryption key;

[0197] Using the encryption key, perform m rounds of encryption on the original company data. Each round uses a different encryption algorithm (such as AES, RSA, Blowfish, Twofish) and encryption key to encrypt the encrypted data obtained in the previous round to obtain new encrypted data, until the mth round, to obtain the final encrypted data.

[0198] The method of intelligently managing encrypted data includes:

[0199] Classify encrypted data and implement a role-based access control (RBAC) system. Set fine-grained permissions to control access to the field level; monitor and record all access attempts and operations in real time, establish a centralized key management system, securely store and manage all encryption keys, and regularly update encryption keys; formulate data retention policies, review and update them regularly, move infrequently used encrypted data to low-cost storage, implement intelligent data sharding and distributed storage, and optimize the processing efficiency of large-scale encrypted data; implement regular backup strategies for encrypted data, and use off-site backup and multi-cloud storage to enhance data reliability and availability, regularly test data recovery processes to ensure rapid data recovery in emergency situations; securely synchronize encrypted data between multiple locations to ensure data consistency.

[0200] This embodiment, by adopting the method of frequent pattern mining and pattern tree fusion, efficiently integrates multi-source heterogeneous data within the company to realize data processing, combines the comprehensive analysis of structured data characteristics and unstructured data information entropy, deeply mines data value, and provides strong support for company decision-making. The system adopts a dynamic encryption mechanism based on data sensitivity and security requirements, and generates encryption keys in combination with data characteristics, which greatly improves data security. The intelligent encrypted data storage and management function optimizes the utilization efficiency of data, helps the company to better realize data-driven, improve operational efficiency, explore new business opportunities, and enhance market competitiveness. In addition, it ensures the compliance of the company's data processing and reduces compliance risks, providing the company with a comprehensive, intelligent and secure data governance solution that can significantly enhance the company's data asset value.

[0201] Embodiment 2

[0202] For the parts not described in detail in this embodiment, please refer to the description of Embodiment 1, which provides a data governance management method, including:

[0203] S1. Collect the original company data from n data sources within the company, and pre-process the original company data to obtain a standard company data set;

[0204] S2. Analyze the standard company data set to obtain structured data features and unstructured data information entropy;

[0205] S3, based on the structured data characteristics and the information entropy of the unstructured data, encrypt the original company data in m rounds to generate encrypted data;

[0206] S4. Store encrypted data and manage the encrypted data intelligently.

[0207] Embodiment 3

[0208] This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the operation mode of the data governance management method provided above is implemented.

[0209] Since the electronic device introduced in this embodiment is an electronic device used to implement a data governance management method in the embodiment of this application, based on the data governance management method introduced in the embodiment of this application, a person skilled in the art can understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of this application is not described in detail here. As long as a person skilled in the art implements an electronic device used in a data governance management method in the embodiment of this application, it belongs to the scope of protection of this application.

[0210] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0211] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical users in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A data governance management system, characterized in that: include: The data collection module is used to collect the original company data from n data sources within the company and pre-process the original company data to obtain a standard company data set; Data analysis module, used to analyze standard company data sets to obtain structured data features and unstructured data information entropy; The encryption module performs m rounds of encryption on the original company data based on the structured data characteristics and the information entropy of the unstructured data to generate encrypted data; A data storage management module is used to store encrypted data and intelligently manage the encrypted data. Each module is connected via wired and / or wireless means; Obtain the frequent patterns of original company data from different data sources, and integrate the original company data based on the frequent patterns to obtain a standard company data set; The frequent pattern acquisition method includes: Clean the original company data and determine the transaction object for each business entity based on the structure of the original company data. If it is row-level data, each row record is regarded as a transaction object. If it is document-level data, each document is regarded as a transaction object. Select all attributes of the transaction object as item sets; all attributes include categorical attributes, numerical attributes, and set attributes; Construct a transaction data set, traverse each transaction object, and for categorical attributes, use the attribute value as the item of the corresponding transaction object in the transaction data set; for numerical attributes, perform discretization to obtain the corresponding discrete value, and map the discrete value to N interval items corresponding to the transaction object in the transaction data set; for set attributes, use each element in the set as the item of the corresponding transaction object in the transaction data set; According to the transaction data set, calculate the support S(i) of each item i of the transaction object T; S(i)=∑(w(T)×r(i,j)); Among them, w(T) is the comprehensive weight of transaction object T, r(i,j) is the association strength between item i and item j; Where |T| is the number of items in the transaction object T; Where, freq(i,j) is the frequency of item i and item j appearing at the same time, sim(i,j) is the semantic similarity score of item i and item j, dist(i,j) is the average distance between item i and item j in transaction object T; α and β are weight hyperparameters; freq(i) is the frequency of item i appearing, freq(j) is the frequency of item j appearing, and max() is the maximum value; Preset the minimum support threshold min_sup, sort all items i whose S(i) is greater than min_sup in descending order according to the size of support, and generate a frequent item header table; construct an FP tree based on the frequent item header table to obtain frequent patterns; The method of performing m-round encryption on the original company data includes: Defines the number of rounds to encrypt raw corporate data Among them, SU is the data sensitivity score, RU is the security requirement score; θ1 is the sensitivity weight, θ2 is the security requirement weight; Data sensitivity score SU = w2×Sp+(1-w2)×Sc; Among them, Sp is the confidentiality score, Sc is the value score, and w2 is the score balance weight parameter; RU=w4×Rl+w5×Rc; Among them, Rl is the risk assessment score, Rc is the company's compliance score, w4 is the risk weight, and w5 is the compliance weight; Generate a first key part based on the structured data feature, and generate a second key part based on the unstructured data information entropy; The first key part is generated by: Sort the local occurrence probabilities of all words, take the values ​​of the first N3 local occurrence probabilities, and concatenate them into a string S1; Sort the word distribution density values ​​of all words, take the first M3 values, and concatenate them into a string S2; Extract the non-zero elements in the diagonal elements, upper triangular elements, or lower triangular elements of the word association information value matrix, and concatenate them into a string S3; Concatenate S1, S2, and S3 into a string S4, and perform a hash operation on S4 to obtain a fixed-length hash value H1, i.e., the first key part; The first key part is generated in the following manner: Take the decimal part of the information entropy of the unstructured data to obtain a decimal X1 between 0 and 1, multiply X1 by a preset integer M5, and take the remainder of M5 to obtain an integer Y1; convert Y1 into a fixed-length string S5, perform a hash operation on S5, and obtain a fixed-length hash value H2, i.e., the second key part; Concatenate the first key part and the second key part to generate an m-round encryption key; Using the encryption key, m rounds of encryption are performed on the original company data. In each round, different encryption algorithms and encryption keys are used to encrypt the encrypted data obtained in the previous round to obtain new encrypted data, until the mth round, the final encrypted data is obtained.

2. A data governance management system according to claim 1, characterized in that: The original company data includes business data, company core data and log data; The business data includes customer information forms, order information forms, product information forms and customer feedback; the company's internal data includes bidding data, employee information forms and internal company documents.

3. A data governance management system according to claim 2, characterized in that: The method of constructing the FP tree based on the frequent item header table to obtain the frequent pattern includes: Create a root node as an empty FP tree. For each transaction object, insert the frequent item header table as a path into the FP tree. Starting from the root node, for each node on the path, if there is a child node that is the same as the current item, increase the child node count. Otherwise, create a new node as a child node and link it to the path of the corresponding node in the frequent item header table. The count of the last node in the path is set to 1. For each node p, traverse the corresponding linked list to obtain the support count L_p of node p; If L_p is greater than the preset counting threshold Tup, then node p forms a frequent pattern of length 1; For each frequent pattern P whose length is greater than 1, traverse all paths associated with P on the FP tree, form the paths prefixed by P into a conditional pattern base, build a conditional FP tree of P based on the conditional pattern base, and recursively mine on the conditional FP tree of P to obtain frequent patterns prefixed by P.

4. A data governance management system according to claim 3, characterized in that: The method of constructing a conditional FP tree of P based on the conditional pattern base includes: For a frequent pattern P, traverse all paths associated with P on the FP tree. Each path is a suffix pattern, denoted as a. All suffix patterns a form a conditional pattern base. Calculate the frequency freq(a) of each suffix pattern a in the conditional pattern base; Filter out the suffix patterns whose frequency freq(a) is less than the preset frequency threshold, and take the remaining suffix patterns as frequent suffix patterns b; Calculate the support count Xb of b, build the path of the conditional FP tree corresponding to b based on Xb, combine the paths of all conditional FP trees, and build the conditional FP tree of P.

5. A data governance management system according to claim 4, characterized in that: The methods for integrating the original company data include: Define a pattern tree. Each frequent pattern corresponds to a pattern tree. The root node of the pattern tree is empty. Each node corresponds to an item in the frequent pattern. The connection relationship between nodes represents the order of items in the pattern. Calculate the structural similarity between different pattern trees. The calculation formula for structural similarity is: Where, MCS(T1,T2) is the number of nodes of the largest common subtree of T1 and T2, |T1| is the number of nodes of pattern tree T1, |T2| is the number of nodes of pattern tree T2; EH(T1,T2) is the structural similarity between pattern tree T1 and pattern tree T2; The similarity between nodes is calculated by the node similarity function, and the weighted sum of the structural similarity and the node similarity is taken as the overall similarity between the pattern trees; A similarity threshold is preset, and all paired pattern trees whose overall similarity is greater than the similarity threshold are regarded as high-similarity pattern pairs; Identify the node pairs on the same level on the high-similarity pattern pairs. For the node pairs on the same level, if they have the same number of child nodes and the relative position order of the child nodes, merge the corresponding node pairs into one node. During the merging process, retain the parallel relationship of the nodes. For high-similarity pattern pairs, if the entity types are the same or have semantic associations, they are fused into one node; the set of pattern trees obtained after the fusion of all high-similarity pattern pairs is used as the unified pattern representation of the original company data, that is, the integration of the original company data is completed to obtain a standard company data set.

6. A data governance management system according to claim 5, characterized in that: The method for obtaining the maximum common subtree of T1 and T2 includes: Represent T1 and T2 as two sets T1 = {t_u} and T2 = {s_v}, where t_u represents a node in T1 and s_v represents a node in T2; Construct a two-dimensional matrix M1, where the element M1[x][y] represents the size of the largest common subtree of two subtrees with t_u and s_v as root nodes; x is the row index of the two-dimensional matrix M1, and y is the column index of the two-dimensional matrix M1; For any element M1[x][y] in the two-dimensional matrix M1, if the u-th node t_u of T1 and the v-th node s_v of T2 have the same label, then M1[x][y]=M1[x-1][y-1]+1; If the signs of t_u and s_v are different, then M1[x][y]=max(M1[x-1][j], M1[x][y-1]); Then the last element M1[m2][n2] of the two-dimensional matrix M1 is the size of the largest common subtree of the two trees T1 and T2; m2 is the last row of the two-dimensional matrix M1, and n2 is the last column of the two-dimensional matrix M1; by backtracing the two-dimensional matrix M1, the structure of the largest common subtree is reconstructed; that is, the acquisition of the largest common subtree is completed.

7. A data governance management system according to claim 6, characterized in that: The method of obtaining the structured data feature includes: Segment the structured data in the standard company data set to obtain F1 words, and calculate the local occurrence probability, word distribution density value, and word association information value matrix of each word; the local occurrence probability, word distribution density value, and word association information value matrix of each word are the structured data features; Count the probability P(R) of the attribute value R of each word; then the local occurrence probability of the word Among them, HG is the total number of data entries, WE(R) is the global weight of the attribute value R; WE(R)=w1×r1(R)+(1-w1)×r2(R); Among them, r1(R) is the value distinguishing ability weight of the attribute value R, r2(R) is the value semantic weight of the attribute value R; w1 is the balance coefficient; Where x_s is the sth value in the range of the attribute value R, p(x_s) is the probability of occurrence of the sth value in the range of the attribute value R; m3 is the total number of values ​​in the range of the attribute value R; Among them, sk(x_s) is the semantic importance score of x_s, sk(x_z) is the semantic importance score of x_z, and x_z is the zth value in the value range of the attribute value R; For each word g, its word distribution density value DF(g) = df(g) × idf(g); Where df(g) is the document frequency of word g in the entire text collection, and idf(g) is the complement of the document frequency of word g in the entire text collection; For any two words g and l, calculate the point mutual information value PMI(g,l) between them; Among them, P(g,l) is the probability of words g and l appearing at the same time, P(g) is the probability of word g appearing alone, and P(l) is the probability of word l appearing alone; Initially, an empty matrix is ​​created, and the words are used as the rows and columns of the empty matrix. The corresponding point mutual information values ​​are used as the corresponding matrix elements, and the word association information value matrix is ​​obtained.

8. A data governance management system according to claim 7, characterized in that: The method of obtaining the information entropy of the unstructured data includes: Convert the unstructured data in the standard company data set into text form to obtain text data; remove stop words from the text data to obtain perfect text data; Convert the perfect text data into a set of points in a high-dimensional vector space, and construct a regular grid to divide the high-dimensional vector space into cubes of size d; Calculate the minimum number of cubes N(d) required to cover the point set; then the calculation formula for the box counting dimension DL is: Under the condition of scale d, obtain the probability p_J(d) of the points contained in each cube J; And calculate the information entropy H(d) = -∑p_J(d)·log(p_J(d)) of the perfect text data at scale d; Based on the information entropy H(d), the information dimension is calculated Get the matrix composed of the distances between points. For a given scale d, count the number of point pairs C(d) whose distance is less than d. Based on the number of point pairs C(d), calculate the correlation dimension The information entropy of unstructured data is Among them, α1, β1 and γ1 are the corresponding weight coefficients; δ is the exponential parameter.

Images