A data asset management system and method

Abstract

The application provides a data asset management system and method, comprising: a data exploration module comprising an exploration script or an exploration program, used for exploring data from a network according to a preset exploration rule, and downloading and storing the data; an asset determination module, used for extracting characteristic attributes of the data, inputting the characteristic data into a pre-trained deep neural network, and outputting value information of the data; a level determination module, used for obtaining a characteristic attribute category, and calculating a level of the data according to the characteristic attribute category and the value information; and a data label module, used for generating a data label according to the attribute type and the value information, and extracting a serial number of a property name of the data corresponding to the byte of the data label to generate a monitoring code. Through classification of data and setting of the monitoring code, the application can detect a data state in real time, realize life cycle monitoring of the data, and ensure safety of the data.

Description

TECHNICAL FIELD

[0001] The present application claims a data management technology, and particularly relates to a data asset management system and method. BACKGROUND

[0002] With the maturity of the Internet, big data and other industries, data assets have become an important asset. However, data assets are intangible and exist in the form of data. In order to improve the transaction value of data assets, data asset management has become an important work content.

[0003] At present, data asset management is still in its early stages, and only realizes the confirmation of data value and the classification of data. Although this data asset management method provides certain convenience for data transaction and use, it completely loses control over data asset management after data transaction is completed, which may lead to data security problems. SUMMARY

[0004] In order to solve the above problems in the prior art, the present application provides a data management technology, and particularly relates to a data asset management system and method.

[0005] The present application provides a data asset management system, comprising:

[0006] A data exploration module comprising an exploration script or an exploration program, configured to explore data from a network according to a preset exploration rule, and download and store the data;

[0007] An asset determination module configured to extract a characteristic attribute of the data, input the characteristic data into a pre-trained deep neural network, and output value information of the data;

[0008] A level determination module configured to obtain a characteristic attribute category, and calculate a level of the data according to the characteristic attribute category and the value information;

[0009] A data label module configured to generate a data label according to the attribute type and the value information, and extract a serial number of a property name of the data corresponding to the byte of the data label according to a byte order of the data label to generate a monitoring code.

[0010] Optionally, the data comprises structured data and unstructured data.

[0011] Optionally, the data exploration from the network comprises address scanning, database table scanning, and file scanning.

[0012] Optionally, the attribute characteristic comprises a name, an address, and biometric data.

[0013] Optionally, the obtaining the characteristic attribute category comprises: reading from an MD5 code of a data packet or a data file.

[0014] The application further provides a data asset management method, comprising:

[0015] According to a preset exploration rule, data is explored from a network, and the data is downloaded and stored;

[0016] Characteristic attributes of the data are extracted, the characteristic data is input into a pre-trained deep neural network, and value information of the data is output;

[0017] The characteristic attribute category is obtained, and a level of the data is calculated according to the characteristic attribute category and the value information;

[0018] According to the attribute type and the value information, a data label is generated, and a monitoring code is generated according to a byte sequence of the data label and a serial number of an attribute name of the data corresponding to the byte.

[0019] Optionally, the data comprises: structured data and unstructured data.

[0020] Optionally, the exploring data from the network comprises: address scanning, database table scanning and file scanning.

[0021] Optionally, the attribute characteristic comprises: a name, an address and biological recognition data.

[0022] Optionally, the obtaining the characteristic attribute category comprises: reading from an MD5 code of a data packet or a data file.

[0023] Compared with the prior art, the application has the following advantages:

[0024] The application provides a data asset management system, comprising: a data exploration module comprising an exploration script or an exploration program, configured to explore data from a network according to a preset exploration rule, and download and store the data; an asset determination module configured to extract characteristic attributes of the data, input the characteristic data into a pre-trained deep neural network, and output value information of the data; a level determination module configured to obtain the characteristic attribute category, and calculate a level of the data according to the characteristic attribute category and the value information; and a data label module configured to generate a data label according to the attribute type and the value information, and generate a monitoring code according to a byte sequence of the data label and a serial number of an attribute name of the data corresponding to the byte. Through classification of the data and setting of the monitoring code, the application can detect a data state in real time, realize life cycle monitoring of the data, and ensure safety of the data. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a schematic diagram of a data asset management system in the present application.

[0026] Figure 2 is a schematic diagram of a data asset management system in the present application.

[0027] Figure 3 is a flow chart of a data asset management method in the present application. DETAILED DESCRIPTION

[0028] The following are examples of specific implementation processes provided for detailed description of the technical solutions claimed in the present application, but the present application can also be implemented in other ways different from the description, and those skilled in the art can implement the present application using different technical means under the guidance of the concept of the present application, so the present application is not limited by the following specific examples.

[0029] The present application provides a data asset management system, comprising: a data exploration module comprising an exploration script or an exploration program, configured to explore data from a network according to a preset exploration rule, and download and store the data; an asset determination module configured to extract a characteristic attribute of the data, input the characteristic data into a pre-trained deep neural network, and output value information of the data; a level determination module configured to obtain a characteristic attribute category, and calculate a level of the data according to the characteristic attribute category and the value information; and a data label module configured to generate a data label according to the attribute type and the value information, and extract a serial number of a byte corresponding to an attribute name of the data to generate a monitoring code according to a byte order of the data label. The present application can detect the state of the data in real time, realize the life cycle monitoring of the data, and ensure the security of the data by classifying the data and setting the detection code.

[0030] Figure 1 is a schematic diagram of a data asset management system in the present application.

[0031] Please refer to Figure 1 The data exploration module 101 comprises an exploration script or an exploration program, configured to explore data from a network according to a preset exploration rule, and download and store the data.

[0032] The data asset management system is arranged in a cloud server, and the data exploration module 101 comprises a receiving and sending unit, a control unit, a running unit, a receiving and sending unit, and a storage unit. The receiving and sending unit is configured to receive a control instruction sent by a user end, the control instruction is processed by the control unit to generate an execution instruction of the running unit, and the running of the control unit is controlled. The running unit calls the exploration script or the exploration program according to the execution instruction to explore data and download and store the data. The data exploration from the network comprises address scanning, database table scanning, and file scanning.

[0033] The data exploration module 101 runs in a cloud server, first receives a start instruction from a user end, establishes a virtual machine after verifying the legality of the start instruction, and then starts the data exploration module 101 in the virtual machine.

[0034] The running steps of the data exploration module 101 include: initialization of the data exploration module 101; receiving a setting instruction, including data exploration setting and data download setting; performing an exploration task according to the setting; and completing the exploration task and downloading data.

[0035] Specifically, the data is downloaded to a data asset storage module, which includes a temporary storage area and a fixed storage area. The size of the temporary storage area can be determined according to the data explored by the data exploration module 101. The fixed storage area is provided with a plurality of data category areas.

[0036] The asset determination module 102 is used to extract the characteristic attributes of the data, input the characteristic data into a pre-trained deep neural network, and output the value information of the data.

[0037] The data is stored in the data asset storage module, including: storing the data explored by the data exploration module 101 in the temporary storage area. When the temporary storage area stores data, the data exploration module 101 is no longer started until the temporary storage area is emptied.

[0038] The asset determination module 102 reads the MD5 code (digest information) of the data from the temporary storage area, and extracts the characteristic attributes of the data according to the MD5 code information.

[0039] Specifically, the characteristic attributes are composed of basic characteristics and sensitive characteristics. The basic characteristics include personal information such as name, address, and biometric data, and the sensitive characteristics include habit and interest information such as consumption and preferences.

[0040] After the data in the temporary storage is read by the asset determination module 102, it will be sorted according to the preset rules to form a data sample set with a fixed format.

[0041] The data sample set is input into a pre-trained deep neural network system for data calculation to obtain the value information of the data.

[0042] Specifically, the deep neural network system is divided into three value optimization layers. Each layer optimizes the data and judges the final data to obtain a value score, and converts the value score into value information.

[0043] The processing steps of the deep neural network system on the data are as follows:

[0044] S201 inputting data;

[0045] S202 judging the data integrity, including checking whether the data is missing according to a preset data item list one by one, then recording the missing items, and determining the value factor of the missing items.

[0046] The data is optimized according to the following formula:

[0047]

[0048] The n is the number of all data attribute characteristic categories in the first layer, and is the number of the attribute characteristic categories screened out in the previous layer in the second and third layers, the L is the data amount of each attribute characteristic, the i is the serial number of the attribute characteristic, and the L max refers to the attribute characteristic with the largest data amount.

[0049] S203 calculating the value of each layer according to the attribute characteristic category screened out in each layer, and the formula is as follows:

[0050]

[0051] The B is the value score, the S is the data amount of each data attribute category in the first, second or third layer, the 1i max , 2i max , and 3i max are the values with the largest data amount in the data attribute categories in the first, second and third layers, and the Q, W and E are the value coefficients of each layer.

[0052] The value score is converted into value information, which can be designed by a person skilled in the art according to the actual value judgment range, for example, the value score is limited in a limited range according to a certain proportion.

[0053] A level determination module 103 is configured to acquire the characteristic attribute category, and calculate the level of the data according to the characteristic attribute category and the value score.

[0054] The attribute characteristic and the value information are respectively input into the level determination module 103, and the level is determined according to a certain rule.

[0055] In the present application, the level determination can include a value assignment unit, which detects the missing condition of each attribute characteristic and respectively assigns different fixed values, for example, if completely missing, the value is 0, if partially missing, the value is 0.5, and if not missing, the value is 1.

[0056] Then and the assignment of the corresponding attribute feature category, all data of the corresponding attribute feature category are extracted, the data amount of the all data is compared with the load data amount, and a missing comparison value is obtained.

[0057] A missing threshold is set, when the missing comparison value is greater than the missing threshold, it is judged that the attribute feature category of the data is worthless, at this time, the value parameter is 0. Otherwise, the value parameter is 1.

[0058] The data with the comprehensive value parameter of 1 is extracted, then the proportion of the value of the attribute feature category in the value information is calculated according to the attribute feature category, then the proportion is divided into grades, and each attribute feature is corresponded to the grades.

[0059] Finally, the grade of the data is determined according to the distribution of the attribute feature category of each grade. For example, the highest value of the attribute feature category in the grade distribution is selected as the data grade, or the grade corresponding to the most value of the attribute feature category in the grade distribution is selected as the data grade.

[0060] The data label module 104 is used for generating a data label according to the attribute type and the value information, and extracting a monitoring code corresponding to the serial number of the attribute name of the data according to the byte order of the data label.

[0061] Specifically, the data label is composed of three parts, including a word head, a position code and a label code, the word head records the attribute feature of the data, the position code refers to the position of the data coding, and the label code refers to the code of data insertion.

[0062] For example, XM 0033 10 indicates that the word head is XM, and the first 33 characters start with 10 characters as the label code.

[0063] The monitoring code is generated according to the label code and the attribute name, for example: name-XM 0033 10, at this time, when the data is processed, the data label module 104 reads the monitoring code, verifies the accuracy of the monitoring code, and sends the processing information of the data to the monitoring platform according to the verification result.

[0064] The application also provides a data asset management method for managing data assets.

[0065] Figure 3 is the data asset management flowchart in the application.

[0066] Please refer to Figure 3 Fig. 1, S301 downloads and stores the data from the network according to the preset exploration rule.

[0067] The data asset management system is arranged in a cloud server, and the data exploration module 101 comprises a receiving and sending unit, a control unit, a running unit, a receiving and sending unit, and a storage unit. The receiving and sending unit is configured to receive a control instruction sent by a user terminal, the control instruction is processed by the control unit to generate an execution instruction of the running unit, and the running of the control unit is controlled. The running unit calls an exploration script or an exploration program according to the execution instruction to perform data exploration and download and storage. The data exploration from the network comprises address scanning, database table scanning, and file scanning.

[0068] The data exploration module 101 runs in the cloud server, first receives a start instruction from a user terminal, verifies the legality of the start instruction, establishes a virtual machine, and then starts the data exploration module 101 in the virtual machine.

[0069] The running steps of the data exploration module 101 comprise: initialization of the data exploration module 101; receiving a setting instruction, including data exploration setting and data download setting; performing an exploration task according to the setting; and completing the exploration task and downloading data.

[0070] Specifically, the data is downloaded to a data asset storage module, the data asset storage module comprises a temporary storage area and a fixed storage area, the size of the data explored by the data exploration module 101 can be determined according to the temporary storage area, and the fixed storage area is provided with a plurality of data category areas.

[0071] Please refer to Figure 3 As shown in the figure, S302 extracts the characteristic properties of the data, inputs the characteristic data into a pre-trained deep neural network, and outputs the value information of the data.

[0072] The data is stored in the data asset storage module, comprising: storing the data explored by the data exploration module 101 in the temporary storage area, when the temporary storage area stores data, the data exploration module 101 is no longer started until the temporary storage area is emptied.

[0073] The asset determination module 102 reads the MD5 code (digest information) of the data from the temporary storage area, and extracts the characteristic properties of the data according to the MD5 code information.

[0074] Specifically, the characteristic properties are composed of basic characteristics and sensitive characteristics, the basic characteristics include personal information such as name, address, and biometric data, and the sensitive characteristics include habit and interest information such as consumption and preference.

[0075] The data of the temporary memory is read by the asset determination module 102, and is arranged according to preset rules to form a data sample set with a fixed format.

[0076] The data sample set is input into a pre-trained deep neural network system for data calculation to obtain value information of the data.

[0077] Specifically, the deep neural network system is divided into three value optimization layers, each layer optimizes the data and judges the final data to obtain a value score, and converts the value score into value information.

[0078] The processing steps of the deep neural network system on the data are as follows:

[0079] Input data;

[0080] Judge the completeness of the data, including comparing whether the data is missing according to a preset data item list one by one, then recording the missing items, and determining the value factor of the missing items.

[0081] Optimize the data according to the following formula:

[0082]

[0083] The n is the number of all data attribute feature categories in the first layer, and is the number of attribute feature categories screened out in the previous layer in the second and third layers, the L is the data amount of each attribute feature, the i is the serial number of the attribute feature, and the L max refers to the attribute feature with the largest data amount.

[0084] According to the attribute feature category screened out in each layer, the value of each layer is calculated, and the formula is as follows:

[0085]

[0086] Among them, the B is the value score, the S is the data amount of each data attribute category in the first, second or third layer, the 1i max , 2i max , 3i max is the value with the largest data amount in the data attribute category in the first, second and third layers, and the Q, W and E are the value coefficients of each layer.

[0087] Based on the conversion of the value score into value information, the actual value judgment range can be designed by those skilled in the art, for example, the value score is limited in a limited range according to the proportion.

[0088] Please refer to Figure 3As shown, S303 obtains the feature attribute category, and calculates the level of the data according to the feature attribute category and the value information.

[0089] The attribute feature and the value information are determined according to certain rules according to the input information of the level determination module 103 respectively.

[0090] In the present application, the level determination can include a value assignment unit, which respectively assigns different fixed values to the missing conditions of each attribute feature, for example, 0 for complete missing, 0.5 for partial missing, and 1 for no missing.

[0091] Then, according to the value assignment, all data corresponding to the attribute feature category are extracted, and a missing comparison value is obtained by comparing the data amount of the all data with the load data amount.

[0092] A missing threshold is set, and when the missing comparison value is greater than the missing threshold, it is judged that the attribute feature category of the data has no value, and the value parameter is 0 at this time. Otherwise, the value parameter is 1.

[0093] The data with the overall value parameter of 1 is extracted, and then the proportion of the value of the category in the value information is calculated according to the attribute feature category, and then the proportion is divided into levels, and each attribute feature is corresponded to the levels.

[0094] Finally, the level of the data is determined according to the distribution of the attribute feature category of each level. For example, the highest value of the attribute feature category in the level distribution is selected as the data level, or the level corresponding to the most value of the attribute feature category in the level distribution is selected as the data level.

[0095] Please refer to Figure 3 As shown, S304 generates a data label according to the attribute type and the value information, and extracts the serial number of the attribute name of the data corresponding to the byte to generate a monitoring code according to the byte order of the data label.

[0096] Specifically, the data label is composed of three parts, including a word head, a position code and a label code, the word head records the attribute feature of the data, the position code refers to the position of the data coding, and the label code refers to the code of data insertion.

[0097] For example: XM 0033 10, which means that the word head is XM, and the first 33 characters start with 10 characters as the label code.

[0098] According to the tag code and attribute name, a monitoring code is generated, for example: name-XM 0033 10, at this time, when the data is processed, the data tag module 104 reads the monitoring code, verifies the accuracy of the monitoring code, and according to the verification result, sends the processing information of the data to the monitoring platform.

[0099] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the exemplary description of the above terms does not necessarily mean the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described can be appropriately combined in any one or more embodiments or examples.

[0100] Although the embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made thereto without departing from the principles and spirit of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. A data asset management system, characterized by, The application relates to a data exploration system and method. The data exploration module comprises an exploration script or an exploration program, which is used for exploring data from a network according to preset exploration rules, downloading and storing the data. The asset determination module is used for extracting feature attributes of the data, inputting the feature attributes into a pre-trained deep neural network, and outputting value information of the data. The deep neural network comprises three value optimization layers, each of which filters attribute features A according to the following formula: ； Wherein, n is the number of all data attribute characteristic categories at the first layer, and is the number of attribute characteristic categories screened out at the previous layer at the second and third layers; L is the data volume of each attribute characteristic, and i is the serial number of the attribute characteristic, refers to the attribute characteristic with the largest data volume. Then, value score B is calculated based on the following formula, and value information of the data is outputted. ； S is the data amount of each data attribute category in the first, second or third layer; 、 、 is the maximum value of the data amount of the data attribute category in the first, second and third layer; Q, W and E are the value coefficients of the first, second and third layer respectively; the level determination module is configured to acquire the feature attribute category, and calculate the level of the data according to the feature attribute category and the value information through missing threshold judgment and attribute category distribution analysis. The data label module is used for generating a data label according to the attribute category and the value information, extracting a monitoring code corresponding to a serial number of the attribute name of the data according to a byte sequence of the data label, and the format of the monitoring code is "word head-position code-label code".

2. The data asset management system of claim 1, wherein, The data comprises structured data and unstructured data.

3. The data asset management system of claim 1, wherein, The data exploration from the network comprises address scanning, database table scanning and file scanning.

4. The data asset management system of claim 1, wherein, The attribute features comprise names, addresses and biological identification data.

5. The data asset management system of claim 1, wherein, The feature attribute category is obtained by reading an MD5 code of a data packet or a data file.

6. A data asset management method characterized by, The application relates to a data exploration system and method. The data exploration module comprises an exploration script or an exploration program, which is used for exploring data from a network according to preset exploration rules, downloading and storing the data. The asset determination module is used for extracting feature attributes of the data, inputting the feature attributes into a pre-trained deep neural network, and outputting value information of the data. The feature attribute category is obtained, and the level of the data is calculated through missing threshold judgment and attribute category distribution analysis according to the feature attribute category and the value information. The data label module is used for generating a data label according to the attribute category and the value information, extracting a monitoring code corresponding to a serial number of the attribute name of the data according to a byte sequence of the data label, and the format of the monitoring code is "word head-position code-label code". The deep neural network comprises three value optimization layers, each of which filters attribute features A according to the following formula: ； Wherein, n is the number of all data attribute characteristic categories at the first layer, and is the number of attribute characteristic categories screened out at the previous layer at the second and third layers; L is the data volume of each attribute characteristic, and i is the serial number of the attribute characteristic, refers to the attribute characteristic with the largest data volume. Then, value score B is calculated based on the following formula, and value information of the data is outputted. ； S is the data volume of each data attribute category in the first, second or third layer; , , is the maximum value of the data volume of the data attribute category in the first, second and third layers; Q, W and E are the value coefficients of the first, second and third layers, respectively.

7. The data asset management method of claim 6, wherein, The data comprises structured data and unstructured data.

8. The data asset management method of claim 6, wherein, The data exploration from the network comprises address scanning, database table scanning and file scanning.

9. The data asset management method of claim 6, wherein, The attribute features comprise names, addresses and biological identification data.

10. The data asset management method of claim 6, wherein, The feature attribute category is obtained by reading an MD5 code of a data packet or a data file.

Images