Data asset completion method, device, equipment and medium

Abstract

The embodiments of the present disclosure provide a data asset completion method, apparatus, device and medium, which relate to the field of big data technology. The method includes: performing domain division on the table metadata assets of the system data assets, obtaining the domain type to which each table metadata asset belongs, and the table metadata assets include table metadata and field metadata; performing integrity assessment on each table metadata asset, marking the missing table metadata assets, and obtaining marked table metadata assets; matching the marked table metadata assets with the data to be matched based on the domain type, and completing the marked table metadata assets based on the matching results, and the data to be matched includes a data asset dictionary. According to the technical solution of the embodiments of the present disclosure, accurate identification of data asset domains and automatic completion of missing data can be achieved, which greatly improves the readability and availability of key information in data assets.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and more specifically, to a data asset completion method, device, electronic device, and computer-readable medium. Background Art

[0002] Data asset management is a hot topic in the current digital transformation of enterprises, giving rise to new industrial forms and business models.

[0003] Currently, automated data asset scanning, collection, and data analysis platforms are available on the market. However, due to the lack of data operational norms and standardization in some data asset management systems, or due to historical cutovers, data loss issues may occur. As a result, enterprise-level data assets generated through cross-collection and integration of various data types also suffer from varying degrees of information loss.

[0004] Therefore, how to complete the key information of the collected data assets is a technical problem that urgently needs to be solved.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0006] The purpose of the embodiments of the present disclosure is to provide a data asset completion method, device, electronic device and computer-readable medium, thereby at least to a certain extent completing the key information of the collected data assets and avoiding the problem of missing key information of data assets.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0008] According to a first aspect of an embodiment of the present disclosure, a data asset completion method is provided, comprising: domaining table metadata assets of system data assets to obtain a domain type to which each table metadata asset belongs, wherein the table metadata assets include table metadata and field metadata; performing a completeness assessment on each of the table metadata assets, marking missing table metadata assets, and obtaining marked table metadata assets; matching the marked table metadata assets with data to be matched based on the domain type, and completing the marked table metadata assets based on the matching result, wherein the data to be matched includes a data asset dictionary, and the data asset dictionary is a dictionary formed by complete table metadata assets.

[0009] According to the first aspect, in some example embodiments, matching the tag table metadata asset with the data to be matched and completing the tag table metadata asset based on the matching result includes: performing string matching on the tag table metadata asset and the data to be matched to determine a corresponding matching result, wherein the matching result includes: one of: an exact match, a keyword match, and a fuzzy match; determining a corresponding matching score based on the matching result using a decision tree model; and completing the tag table metadata asset based on the matching score.

[0010] According to the first aspect, in some example embodiments, determining the corresponding matching score based on the matching result using a decision tree model includes: determining the matching category corresponding to the matching result using a decision tree model, and determining a first matching score corresponding to the matching category; determining the matched keyword and the offset corresponding to the keyword, the offset including a system offset, a table offset, and a field offset; and determining a second matching score based on the first matching score, the system offset, the table offset, and the field offset.

[0011] According to the first aspect, in some example embodiments, the completing the tag table metadata asset based on the matching score includes: determining the matching data asset corresponding to the tag table metadata asset based on the matching score; generating supplementary information based on semantic information of the matching data asset, and completing the tag table metadata asset.

[0012] According to the first aspect, in some example embodiments, the table metadata assets of the system data assets are divided into domains to obtain the domain type to which each table metadata asset belongs, including: vectorizing the fields of the table structure of the table metadata assets of the system data assets; obtaining the central word of the table structure of the table metadata assets based on a bag-of-words model; obtaining the central topic of the table structure of the table metadata assets based on a topic model algorithm; matching the central topic of the table structure of each of the table metadata assets with the topic words of each domain to obtain the domain type to which each of the table metadata assets belongs.

[0013] According to the first aspect, in some example embodiments, the integrity of each of the table metadata assets is evaluated, and table metadata assets with missing items are marked, including: determining whether field metadata of each of the table metadata assets is missing; and marking table metadata assets with missing items.

[0014] According to the first aspect, in some example embodiments, the method further includes: extracting keywords and master data from the unlabeled complete table metadata asset according to the domain type to which the table metadata asset belongs, and establishing the data asset dictionary based on the keywords and the master data.

[0015] According to a second aspect of an embodiment of the present disclosure, a data asset completion device is provided, comprising: a domain division module for performing domain division on table metadata assets of system data assets to obtain a domain type to which each table metadata asset belongs, wherein the table metadata assets include table metadata and field metadata; a completeness assessment module for performing completeness assessment on each of the table metadata assets, marking missing table metadata assets, and obtaining marked table metadata assets; a completion processing module for matching the marked table metadata assets with data to be matched based on the domain type, and performing completion processing on the marked table metadata assets based on the matching result, wherein the data to be matched includes a data asset dictionary, and the data asset dictionary is a dictionary formed by complete table metadata assets.

[0016] According to the second aspect, in some example embodiments, the completion processing module includes: a matching unit, used to perform string matching on the tag table metadata asset and the data to be matched, and determine a corresponding matching result, wherein the matching result includes: one of: a complete match, a keyword match, and a fuzzy match; a matching score determination unit, used to determine a corresponding matching score based on the matching result using a decision tree model; and a completion unit, used to perform completion processing on the tag table metadata asset based on the matching score.

[0017] According to the second aspect, in some example embodiments, the matching score determination unit is further used to: use a decision tree model to determine the matching category corresponding to the matching result, and determine a first matching score corresponding to the matching category; determine the matched keyword and the offset corresponding to the keyword, the offset including a system offset, a table offset, and a field offset; and determine a second matching score based on the first matching score, the system offset, the table offset, and the field offset.

[0018] According to the second aspect, in some example embodiments, the completion unit is further used to: determine the matching data asset corresponding to the tag table metadata asset based on the matching score; generate supplementary information based on semantic information of the matching data asset, and complete the tag table metadata asset.

[0019] According to the second aspect, in some example embodiments, the domain sub-module is further used to: vectorize the fields of the table structure of the table metadata assets of the system data assets; obtain the central word of the table structure of the table metadata assets based on the bag-of-words model; obtain the central topic of the table structure of the table metadata assets based on the topic model algorithm; match the central topic of the table structure of each of the table metadata assets with the topic words of each domain to obtain the domain sub-type to which each of the table metadata assets belongs.

[0020] According to the second aspect, in some example embodiments, the completeness assessment module is further configured to: determine whether field metadata of each of the table metadata assets is missing; and mark the table metadata assets that are missing.

[0021] According to the second aspect, in some example embodiments, the device further includes: an extraction module for extracting keywords and master data from unlabeled complete table metadata assets according to the domain type to which the table metadata assets belong; and a dictionary establishment module for establishing the data asset dictionary based on the keywords and the master data.

[0022] According to a third aspect of an embodiment of the present disclosure, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the data asset completion method as described in the first aspect of the above embodiment is implemented.

[0023] According to the fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the data asset completion method as described in the first aspect of the above embodiment.

[0024] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0025] In some embodiments of the present disclosure, on the one hand, the table metadata assets of the system data assets are domained to obtain the domain type to which each table metadata asset belongs, and the domain type to which the table metadata assets of each system belong can be accurately obtained; on the other hand, the integrity of each table metadata asset is evaluated, and the missing table metadata assets are marked, so that the missing table metadata assets can be accurately determined; on the other hand, based on the domain type, the marked table metadata assets are matched with the data to be matched, and the marked table metadata assets are completed based on the matching results, so that key information of the missing data assets can be extracted and completed, thereby realizing accurate identification of data asset domains and automatic completion of missing data, greatly improving the readability and availability of key information in data assets, and enhancing the value of data sharing.

[0026] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0028] Figure 1 A flowchart of a data asset completion method according to some example embodiments of the present disclosure is shown;

[0029] Figure 2 A schematic diagram of a process for partitioning data assets according to some example embodiments of the present disclosure is shown;

[0030] Figure 3 A schematic diagram of a process for completing data assets according to some example embodiments of the present disclosure is shown;

[0031] Figure 4 A flowchart illustrating a data asset completion method according to some further example embodiments of the present disclosure is shown;

[0032] Figure 5 A schematic diagram showing a process of partitioning data assets according to some further exemplary embodiments of the present disclosure is shown.

[0033] Figure 6 A schematic diagram of a process for completing data assets according to some other example embodiments of the present disclosure is shown;

[0034] Figure 7 A schematic diagram illustrating a decision tree model according to some example embodiments of the present disclosure is shown;

[0035] Figure 8 A schematic structural diagram of a data asset completion device according to an embodiment of the present disclosure is shown;

[0036] Figure 9 A schematic structural diagram of an electronic device in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0038] In addition, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present disclosure.

[0039] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0040] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0041] Hereinafter, the technical solutions in the exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0042] Figure 1 A schematic flow chart illustrating a data asset completion method according to some example embodiments of the present disclosure is provided. The data asset completion method provided in these examples can be executed by a computing device with computing capabilities, such as a desktop computer. The data asset completion method includes steps S110 to S130. The data asset completion method in these example embodiments is described in detail below with reference to the accompanying figures.

[0043] refer to Figure 1 As shown, in step S110, the table metadata assets of the system data assets are divided into domains to obtain the domain type to which each table metadata asset belongs. The table metadata assets include table metadata and field metadata.

[0044] In an exemplary embodiment, there are multiple system data assets, each of which includes multiple table metadata assets, each of which includes table metadata and field metadata. For example, let the system data asset set P = {P1, P2, ..., Pt, ..., Pm}, where Pt is the t-th system data asset and m is the total number of systems. System data asset P(x) = {T1, T2, ..., Tt, ..., Tn}, where Tt is the t-th table metadata asset and n is the total number of table metadata in the P(x) system. Table metadata asset T(x) = {C1, C2, ..., Ct, ..., Ck}, where Ct is the t-th field metadata asset and k is the total number of field metadata in the T(x) table.

[0045] Furthermore, the fields of the table structure of the table metadata assets of the system data assets are vectorized; the central word of the table structure of the table metadata assets is obtained based on the bag-of-words model; the central theme of the table structure of the table metadata assets is obtained based on the topic model algorithm; the central theme of each table structure is matched with the topic word of each domain to obtain the sub-domain type to which each table metadata asset belongs. For example, the data assets of telecommunications enterprises can be divided into the following domains according to the domain type to which the data assets belong: IT (BSS (Business Support System), OSS (Operation Support System), MSS (Management Support System), EDA (Enterprise Data Application), ITM (IT Management)), cloud & network (wireless network, core network, transmission network, data network, cloud, IDC (Internet Data Center), dynamic environment, integrated support and scheduling), business network or platform (payment, cloud resource business platform, video conferencing, smart home, Internet of Things, VoLTE (Voice over Long-Term Evolution), high definition, SMS gateway), security (cloud, network, data) and other domains.

[0046] In step S120 , the integrity of each table metadata asset is evaluated, and missing table metadata assets are marked to obtain marked table metadata assets.

[0047] In an example embodiment, it is determined whether field metadata of each table metadata asset is missing; the missing table metadata assets are marked to obtain marked table metadata assets. For example, the metadata of each field of the table metadata asset Ti is traversed. If the data corresponding to the field metadata of the table metadata asset Ti exists, the table metadata asset Ti and the missing fields are marked.

[0048] Furthermore, the field metadata may include key metadata information such as field name, field type, field length, service description, calling party and calling party.

[0049] In step S130, based on the domain type, the marked table metadata asset is matched with the data to be matched, and the marked table metadata asset is completed based on the matching result. The data to be matched includes a data asset dictionary, which is a dictionary formed by the complete table metadata asset.

[0050] In an example embodiment, based on the domain type, to-be-matched data within the same domain type is determined, the tag table metadata asset is matched with the to-be-matched data, and the tag table metadata asset is completed based on the matching result. For example, assuming the classification type of the tag table metadata asset is Operation-Monitoring & Alarm, the complete table metadata asset within the Operation-Monitoring & Alarm domain is obtained as the to-be-matched data, and a string match is performed between the tag table metadata asset and the to-be-matched data to determine the corresponding matching result, which includes: an exact match, a keyword match, and a fuzzy match; a decision tree model is used based on the matching result to determine the corresponding matching score; and the tag table metadata asset is completed based on the matching score.

[0051] Furthermore, in some example embodiments, the tag table metadata asset is completed based on the matching score, including: determining the matching data asset corresponding to the tag table metadata asset based on the matching score; generating supplementary information based on the semantic information of the matching data asset, and completing the tag table metadata asset.

[0052] according to Figure 1The technical solution in the example embodiment, on the one hand, domains the table metadata assets of the system data assets, obtains the domain type to which each table metadata asset belongs, and can accurately obtain the domain type to which the table metadata assets of each system belong; on the other hand, the integrity of each table metadata asset is evaluated, and the missing table metadata assets are marked, so that the missing table metadata assets can be accurately determined; on the other hand, based on the domain type, the marked table metadata assets are matched with the data to be matched, and the marked table metadata assets are completed based on the matching results, so that key information of the missing data assets can be extracted and completed, thereby realizing accurate identification of data asset domains and automatic completion of missing data, greatly improving the readability and availability of key information in data assets, and enhancing the value of data sharing.

[0053] Figure 2 A schematic diagram of a process for partitioning data assets according to some embodiments of the present disclosure is shown.

[0054] Reference Figure 2 As shown, in step S210, a data asset set is obtained.

[0055] In this example embodiment, let system data asset set P = {P1, P2, ..., Pt, ..., Pm}, where Pt is the tth system asset and m is the total number of system assets. System data asset P(x) = {T1, T2, ..., Tt, ..., Tn}, where Tt is the tth table metadata asset and n is the total number of table metadata in system P(x). Table metadata asset T(x) = {C1, C2, ..., Ct, ..., Ck}, where Ct is the tth field metadata asset and k is the total number of field metadata in table T(x).

[0056] In step S220 , each field of the table structure of the metadata asset is segmented using a word segmentation algorithm and stop words are removed.

[0057] In step S230 , the fields of the table structure are vectorized using one-hot encoding, and then the bag-of-words model is used to obtain the central word in the table structure.

[0058] In the example embodiment, the bag-of-words model is taken as a CBOW (Continuous Bag-Of-Words) model, and the field metadata T(x) of the metadata asset is set to be w input , predict the probability of the central word appearing is w t , the corresponding mathematical expression is p(w t |w input ), as shown in the following formula (1):

[0059] p(w t |w input)=w j =exp(w j ) / ∑exp(w j′ ) (1)

[0060] Among them, p(w t |w input ) to give him the field metadata w input The probability of the central word appearing under the condition of , wj represents the j-th field vector.

[0061] Furthermore, the maximum conditional probability is used to obtain the maximum probability of the table structure center word of each table metadata asset as follows (2):

[0062] maxp(w t |w input )=maxlogw j =w j -log∑exp(w j ) (2)

[0063] Among them, max p(w t |w input ) represents the maximum probability of the central word of the table structure of the table metadata asset.

[0064] In step S240 , after obtaining the central word of each table structure, the central topic of the table structure is obtained using the topic model.

[0065] In the exemplary embodiment, the PLSA (Probabilistic Latent Semantic Analysis) topic model is taken as an example, and p(P n ) to select a system P from the system data asset set P m , with p(T n |P n ) to select a table metadata T from the system data assets P(x) n , with p(C k |T n ) to select a field metadata C from the table metadata asset T(x) k , establish the likelihood function as shown in the following formula (3):

[0066]

[0067] Where Pm is the system data asset, Ck is the field metadata in the table metadata asset, and n(Pm, Ck) is the number of occurrences of the (Pm, Ck) pair. To distinguish them, M' and K' are used to identify the number of corresponding table metadata and field metadata. Taking the logarithm of both sides of equation (3) above, we obtain the following equation (4):

[0068]

[0069] Based on the above formula (4), the probability of the central topic to which the central word of the table structure belongs can be obtained.

[0070] In step S250 , the central theme of the table structure of each table metadata asset is matched with the subject words of each domain type to implement domain division of each table metadata asset.

[0071] In an example embodiment, multiple domain-specific keywords are obtained, and the central theme of the table structure of each table metadata asset is matched with the keyword of each domain-specific type to achieve domain-specific classification of each table metadata asset. For example, multiple domain types are assumed to include IT (BSS, OSS, MSS, EDA, ITM), Cloud & Network (Wireless Network, Core Network, Transmission Network, Data Network, Cloud, IDC, Dynamic Environment, Integrated Support and Scheduling), Business Network or Platform (Payment, Cloud Resource Business Platform, Video Conferencing, Smart Home, Internet of Things, VoLTE, HD, SMS Gateway), Security (Cloud, Network, Data), etc. The central theme of the table structure of the table metadata asset is matched with each domain-specific type, and the matched domain-specific type is used as the domain-specific type of the table metadata asset.

[0072] according to Figure 2 The technical solution in the example embodiment utilizes data asset vectorization technology, combined with one-hot encoding technology, bag-of-words model, and topic model algorithm to achieve accurate identification of domain types of data assets.

[0073] Figure 3 A schematic diagram of a process for completing data assets according to some example embodiments of the present disclosure is shown.

[0074] like Figure 3 As shown, in step S310 , the tag table metadata asset is extracted.

[0075] In step S320 , an asset dictionary is created for the unlabeled complete table metadata assets according to the domains to which the assets belong.

[0076] In step S330 , a rule engine is created based on the asset dictionary, and the rule engine is configured with multiple asset matching rules.

[0077] In an example embodiment, a rule engine that can be linked with data assets is implemented, including metadata management based on keywords and dictionaries and metadata based on regular matching, and a rule system combining dictionary and regular superposition is created to support complex asset matching rules.

[0078] In step S340 , the tag table metadata assets are matched with the data to be matched based on the rule engine.

[0079] In the example embodiment, the asset matching rules customized on demand by the rule engine are used to dynamically build a corresponding rule scanning engine to perform character set checking on the data assets marked as missing and convert them into UTF8 encoding;

[0080] For data assets marked as missing, a string matching algorithm is used to detect keywords. Based on the keyword detection results, a decision tree model is used to calculate the dictionary matching results. For data assets marked as missing, a regular expression matching algorithm is used to detect master data. For master data with a certain verification algorithm, the verifier detection engine is called.

[0081] In step S350, for successful keyword matching results, the specific matched keyword and the keyword's offset in the content to be scanned are recorded; for successful master data regular matching results, the specific matched content and the offset of the content in the content to be scanned are recorded.

[0082] In step S360, based on the dynamic decision tree algorithm, on the basis of the matching results of the asset dictionary and the regular expression, the combination rule is calculated to obtain the matching score, and based on the semantic information of the matched data assets, the supplementary recording information is generated to perform asset supplementary recording.

[0083] In an example embodiment, a decision tree model is used to determine a corresponding match score based on the matching result; and completion processing is performed on the tag table metadata asset based on the matching score. Furthermore, the decision tree model is used to determine a corresponding match score based on the matching result, including: using the decision tree model to determine a match category corresponding to the matching result, where the match categories include exact match, keyword match, and fuzzy match, and determining a first match score corresponding to the match category; determining a matched keyword and an offset corresponding to the keyword, where the offset includes a system offset, a table offset, and a field offset; and determining a second match score based on the first match score, the system offset, the table offset, and the field offset.

[0084] according to Figure 3 The technical solution in the example embodiment, on the one hand, builds an asset dictionary based on the complete data assets, and matches the tag table metadata assets based on the asset dictionary and regular expressions, which can accurately match the complete data assets corresponding to the tag table metadata assets; on the other hand, based on the domain type, the tag table metadata assets are matched with the data to be matched, and the tag table metadata assets are completed based on the matching results, which can extract and complete the key information of the missing data assets, realize the accurate identification of data asset domains and the automatic completion of missing data, greatly improve the readability and availability of key information in data assets, and enhance the value of data sharing.

[0085] Figure 4A flowchart of a data asset completion method according to some other example embodiments of the present disclosure is shown.

[0086] Reference Figure 4 As shown, in step S410, data assets are divided into domains.

[0087] In an exemplary embodiment, partitioning data assets into different domains includes the following steps S501 to S505:

[0088] Step S501 : Scan metadata of each system and aggregate the metadata of each system into an input system data asset set.

[0089] Among them, the system data asset set P = {core processing, collection system, capability opening portal, operation management system, ...}, the system data asset P (operation management system) = {alarm_person, ivr_log, menu_role, ...}, and the table metadata asset T (alarm_person) = {ID, GROUP_ID, ALARM_ID, PERSON_CODE, PERSON_NAME, PHONE_NO, LAN_ID, TYPE, STATUS, D_PERSON_ID, BF_COL1, BF_COL2}.

[0090] Step S502 : Segment each field of the table structure of the metadata asset using a word segmentation algorithm and remove stop words.

[0091] In this example embodiment, taking the table metadata asset set T(alarm_person) = {ID, GROUP_ID, ALARM_ID, PERSON_CODE, PERSON_NAME, PHONE_NO, LAN_ID, TYPE, STATUS, D_BF1, D_BF2, D_BF3}, the "_" in all fields, the obsolete fields marked with "D_", and the spare fields marked with "BF_" are removed from the asset set, and the spare fields marked with "__" are replaced with spaces. The result after removal is: table metadata asset set T(alarm_person) = {ID, GROUP ID, ALARM ID, PERSON CODE, PERSON NAME, PHONE NO, LANID, TYPE, STATUS}, as shown in Table 1 below:

[0092] Table 1: Schematic diagram of the table structure of the metadata asset T (alarm_person)

[0093] Table metadata Table domain Field metadata Field Domain alarm_person Operations > Monitoring & Alerting id System > Encoding alarm_person Operations > Monitoring & Alerting group_id System > Group > Encoding alarm_person Operations > Monitoring & Alerting alarm_id Operations > Monitoring & Alerting > Coding alarm_person Operations > Monitoring & Alerting person_code User > User ID alarm_person Operations > Monitoring & Alerting person_name User>Name alarm_person Operations > Monitoring & Alerting phone_no User>Number alarm_person Operations > Monitoring & Alerting lan_id Region>Code alarm_person Operations > Monitoring & Alerting type System > Type alarm_person Operations > Monitoring & Alerting status System > Programs > Status

[0094] In step S503, the fields of the table structure are vectorized using one-hot encoding, and then the bag-of-words model is used to obtain the central word in the table structure.

[0095] In the example embodiment, let the field metadata T(x) of the table metadata asset be w input , predict the probability of the central word appearing is w t , the corresponding mathematical expression is p(w t |w input ), as shown in the following formula (1):

[0096] p(w t |w input )=w j =exp(w j ) / ∑exp(w j′ ) (1)

[0097] Among them, p(w t |w input ) to give him the field metadata w input The probability of the central word appearing under the condition of , wj represents the j-th field vector.

[0098] Furthermore, the maximum conditional probability is used to obtain the maximum probability of the table structure center word of each table metadata asset as follows (2):

[0099] maxp(w t |w input )=maxlogw j =w j -log∑exp(w j ) (2)

[0100] Among them, maxp(w t |w input ) represents the maximum probability of the central word of the table structure of the table metadata asset.

[0101] Based on the above formula, we can obtain T(alarm_person)={ALARM ID, GROUP ID, LAN ID}.

[0102] In addition, in the example embodiment, the predicted structure is verified. For example, the predicted central word is verified with the table metadata. The one with higher similarity is judged as the accurate prediction of the central word, which can improve the accuracy of domain segmentation. The one with lower similarity is subjected to tuning training and manual labeling.

[0103] In step S504, after obtaining the central word of each table structure, the central topic of the table structure is obtained using the topic model.

[0104] In the exemplary embodiment, the PLSA (Probabilistic Latent Semantic Analysis) topic model is taken as an example, and p(P n ) to select a system P from the system data asset set P m , with p(T n |P n ) to select a table metadata T from the system data assets P(x) n , with p(C k |T n ) to select a field metadata C from the table metadata asset T(x) k , establish the likelihood function as shown in the following formula (3):

[0105]

[0106] Where Pm is the system data asset, Ck is the field metadata in the table metadata asset, and n(Pm, Ck) is the number of occurrences of the (Pm, Ck) pair. To distinguish them, M' and K' are used to identify the number of corresponding table metadata and field metadata. Taking the logarithm of both sides of equation (3) above, we obtain the following equation (4):

[0107]

[0108] Based on the above formula (4), we can get the probability of the central topic to which the central word of the table structure belongs. For example, T(alarm_person) = {monitoring & alarm: 0.825, encoding: 0.13, user identification: 0.05...}

[0109] In step S505 , the central theme of the table structure of each table metadata asset is matched with the subject words of each domain type to implement domain division of each table metadata asset.

[0110] Next, in step S420, a data asset integrity assessment is performed.

[0111] In an exemplary embodiment, assets are aggregated into an input asset set P = {core processing, acquisition system, capability opening portal, operation management system, ...}, a system asset level P (operation management system) = {alarm_person, ivr_log, menu_role, ...}, and a table metadata asset T (alarm_person) = {ID, GROUP_ID, ALARM_ID, PERSON_CODE, PERSON_NAME, PHONE_NO, LAN_ID, TYPE, STATUS, D_PERSON_ID, BF_COL1, BF_COL2}.

[0112] Furthermore, the table metadata asset set T(x) is compared for completeness. Specifically, the comparison content includes key metadata information such as field name, field type, field length, business description, and reference master data. If there is a null value, the field metadata is marked.

[0113] In step S430, intelligent data asset completion is performed.

[0114] In an example embodiment, based on the domain type, the tag table metadata assets are matched with the data to be matched, and the tag table metadata assets are completed based on the matching results. The detailed process of performing intelligent data asset completion may include the following steps S601 to S606:

[0115] S601: Extract and save the metadata of the fields marked as missing.

[0116] S602: Extract and save the metadata of the fields that are not fully marked, and create an asset dictionary.

[0117] S603. Create matching rules. The matching rules include field name exact matching, keyword matching, and fuzzy query matching.

[0118] In an exemplary embodiment, different types of matches are assigned different scores, as shown in Table 2 below:

[0119] Table 2: Matching scores for different matching types

[0120] Match Type Scoring Exact match 10 Keyword matching 3 Fuzzy matching 1

[0121] S604: Generate matching results through the scanning engine according to the established scanning rules and customized asset matching rules.

[0122] In this example embodiment, the person_name field of the alarm_person table of the operation system is missing the Chinese name of the field. The matching results are excerpted from Table 3 below:

[0123] Table 3: Matching results of different fields in the alarm_person table

[0124]

[0125] S605: Adjust the scoring result based on the matching result and the domain.

[0126] In this example embodiment, 6 points are deducted for inconsistencies in the system domain, 4 points are deducted for inconsistencies in the table domain, and 2 points are deducted for inconsistencies in the field domain. An excerpt of the adjusted content of the person_name field in the alarm_person table of the operation system is shown in Table 4 below:

[0127] Table 4: Adjusted matching scores for each field in the alarm_person table

[0128]

[0129]

[0130] S606: Summarize the scores based on the decision tree to obtain the highest score, extract the semantic information of the data asset based on the highest score, generate supplementary information, and complete the asset.

[0131] In an exemplary embodiment, referring to Figure 7 As shown, based on the decision tree, missing assets are associated using three methods: exact match, keyword match, and fuzzy match. If an exact match is found, it is selected and the next decision is made. Offsets for the system, table, and field are compared to determine a final score. Based on the highest score, semantic information for the corresponding data asset is extracted, and supplementary information is generated to complete the asset.

[0132] according to Figures 5 to 7 The technical solution in the example embodiment, on the one hand, utilizes data asset vectorization technology, combined with one-hot encoding technology, bag-of-words model, and topic model algorithm to achieve accurate identification of the domain types of data assets; on the other hand, it can extract and complete key information of the acquired missing data assets, achieve accurate identification of data asset domains and automatic completion of missing data, greatly improve the readability and availability of key information in data assets, and enhance the value of data sharing.

[0133] It should be noted that the above figures are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0134] The following introduces an embodiment of the device disclosed herein, which can be used to execute the above-mentioned data asset completion method of the present disclosure.

[0135] Figure 8 A structural diagram of a data asset completion device according to an embodiment of the present disclosure is shown.

[0136] Reference Figure 8As shown, the data asset completion device 800 includes: a domain division module 810, which is used to divide the table metadata assets of the system data assets into domains and obtain the domain type to which each table metadata asset belongs, and the table metadata assets include table metadata and field metadata; an integrity assessment module 820, which is used to perform integrity assessment on each of the table metadata assets, mark missing table metadata assets, and obtain marked table metadata assets; a completion processing module 830, which is used to match the marked table metadata assets with the data to be matched based on the domain type, and complete the marked table metadata assets based on the matching results, and the data to be matched includes a data asset dictionary, and the data asset dictionary is a dictionary formed by complete table metadata assets.

[0137] In some example embodiments, the completion processing module 830 includes: a matching unit, used to perform string matching on the tag table metadata asset and the data to be matched, and determine a corresponding matching result, wherein the matching result includes: one of: a complete match, a keyword match, and a fuzzy match; a matching score determination unit, used to determine a corresponding matching score based on the matching result using a decision tree model; and a completion unit, used to perform completion processing on the tag table metadata asset based on the matching score.

[0138] In some example embodiments, the matching score determination unit is further used to: use a decision tree model to determine the matching category corresponding to the matching result, and determine a first matching score corresponding to the matching category; determine the matched keyword and the offset corresponding to the keyword, the offset including a system offset, a table offset, and a field offset; and determine a second matching score based on the first matching score, the system offset, the table offset, and the field offset.

[0139] In some example embodiments, the completion unit is further configured to: determine a matching data asset corresponding to the tag table metadata asset based on the matching score; generate supplementary information based on semantic information of the matching data asset, and complete the tag table metadata asset.

[0140] In some example embodiments, the domain sub-module 810 is further used to: vectorize the fields of the table structure of the table metadata assets of the system data assets; obtain the central word of the table structure of the table metadata assets based on the bag-of-words model; obtain the central topic of the table structure of the table metadata assets based on the topic model algorithm; match the central topic of the table structure of each of the table metadata assets with the topic words of each domain to obtain the domain sub-type to which each of the table metadata assets belongs.

[0141] In some example embodiments, the completeness assessment module 820 is further configured to: determine whether field metadata of each of the table metadata assets is missing; and mark the table metadata assets that are missing.

[0142] In some example embodiments, the device 800 further includes: an extraction module for extracting keywords and master data from unlabeled complete table metadata assets according to the domain type to which the table metadata assets belong; and a dictionary building module for building the data asset dictionary based on the keywords and the master data.

[0143] Since the various functional modules of the data asset completion device in the example embodiment of the present disclosure correspond to the steps of the example embodiment of the data asset completion method described above, for details not disclosed in the network device embodiment of the present disclosure, please refer to the embodiment of the data asset completion method described above in the present disclosure.

[0144] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0145] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0146] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0147] In exemplary embodiments of the present disclosure, a computer storage medium capable of implementing the aforementioned method is also provided. A program product capable of implementing the aforementioned method is stored on the computer storage medium. In some possible embodiments, various aspects of the present disclosure may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present disclosure.

[0148] The program product may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0149] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0150] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0151] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0152] The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0153] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0154] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0155] Refer to the following Figure 9 hereinafter, an electronic device 900 according to this embodiment of the present disclosure is described. Figure 9 The electronic device 900 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0156] like Figure 9 As shown, electronic device 900 is implemented as a general-purpose computing device. Components of electronic device 900 may include, but are not limited to, at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting various system components (including storage unit 920 and processing unit 910).

[0157] The storage unit stores program codes, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps described in the "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 910 can perform the following steps: Figure 1As shown in: Step S110, the table metadata assets of the system data assets are divided into domains to obtain the domain type to which each table metadata asset belongs. The table metadata assets include table metadata and field metadata. Step S120, the integrity of each table metadata asset is evaluated, and missing table metadata assets are marked to obtain marked table metadata assets. Step S130, based on the domain type, the marked table metadata assets are matched with the data to be matched, and the marked table metadata assets are completed based on the matching results. The data to be matched includes a data asset dictionary, and the data asset dictionary is a dictionary formed by complete table metadata assets.

[0158] Exemplarily, the processing unit 910 may also execute the data asset completion method in the above embodiment.

[0159] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 9201 and / or a cache memory unit 9202 , and may further include a read-only memory unit (ROM) 9203 .

[0160] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0161] Bus 930 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0162] The electronic device 900 can also communicate with one or more external devices 990 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 900, and / or any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 950. Furthermore, the electronic device 900 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 via a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 900, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0163] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0164] Furthermore, the figures above are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0165] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

Claims

1. A data asset completion method, characterized in that: include: Domaining table metadata assets of system data assets to obtain domain types to which each table metadata asset belongs, wherein the table metadata assets include table metadata and field metadata; The system data assets are telecommunication enterprise data assets; Performing integrity assessment on each of the table metadata assets, marking missing table metadata assets, and obtaining marked table metadata assets; Based on the domain type, the marked table metadata asset is string-matched with the data to be matched to determine a corresponding matching result, wherein the matching result includes: one of: a complete match, a keyword match, and a fuzzy match; a decision tree model is used to determine a corresponding matching score based on the matching result; and the marked table metadata asset is completed based on the matching score; the data to be matched includes a data asset dictionary, which is a dictionary formed by complete table metadata assets; the determining of the corresponding matching score based on the matching result using a decision tree model includes: using a decision tree model to determine a matching category corresponding to the matching result, and determining a first matching score corresponding to the matching category; determining a matched keyword and an offset corresponding to the keyword, wherein the offset includes a system offset, a table offset, and a field offset; and determining a second matching score based on the first matching score, the system offset, the table offset, and the field offset; The table metadata assets of the system data assets are divided into domains to obtain the domain type to which each table metadata asset belongs, including: Vectorize the fields of the table structure of the table metadata assets of the system data assets; Obtaining a central word of the table structure of the table metadata asset based on a bag-of-words model; Based on a topic model algorithm, a central topic of the table structure of the table metadata asset is obtained; The central theme of the table structure of each of the table metadata assets is matched with the subject words of each domain to obtain the sub-domain type to which each of the table metadata assets belongs.

2. The method according to claim 1, characterized in that The completing the tag table metadata asset based on the matching score includes: Determining a matching data asset corresponding to the tag table metadata asset based on the matching score; Supplementary information is generated based on the semantic information of the matching data asset, and the tag table metadata asset is supplemented.

3. The method according to claim 1 or 2, characterized in that Perform integrity assessment on each of the table metadata assets and mark missing table metadata assets, including: Determining whether field metadata of each of the table metadata assets is missing; Mark missing table metadata assets.

4. The method according to claim 3, characterized in that The method further comprises: Extract keywords and master data from unlabeled complete table metadata assets according to the domain type to which the table metadata assets belong; The data asset dictionary is established based on the keywords and the master data.

5. A data asset completion device, characterized in that: include: A domain division module is used to divide table metadata assets of system data assets into domains and obtain the domain type to which each table metadata asset belongs. The table metadata assets include table metadata and field metadata. The system data assets are telecommunication enterprise data assets; An integrity assessment module, configured to perform integrity assessment on each of the table metadata assets, mark missing table metadata assets, and obtain marked table metadata assets; A completion processing module is configured to perform string matching on the tagged table metadata asset and the data to be matched based on the domain type, and determine a corresponding matching result, wherein the matching result includes: one of: a complete match, a keyword match, and a fuzzy match; determine a corresponding matching score based on the matching result using a decision tree model; and perform completion processing on the tagged table metadata asset based on the matching score; the data to be matched includes a data asset dictionary, wherein the data asset dictionary is a dictionary formed by complete table metadata assets; the determining of the corresponding matching score based on the matching result using a decision tree model is configured to: determine a matching category corresponding to the matching result using a decision tree model, and determine a first matching score corresponding to the matching category; determine a matched keyword and an offset corresponding to the keyword, wherein the offset includes a system offset, a table offset, and a field offset; and determine a second matching score based on the first matching score, the system offset, the table offset, and the field offset; The table metadata assets of the system data assets are divided into domains to obtain the domain type to which each table metadata asset belongs, which is configured as follows: Vectorize the fields of the table structure of the table metadata assets of the system data assets; Obtaining a central word of the table structure of the table metadata asset based on a bag-of-words model; Based on a topic model algorithm, a central topic of the table structure of the table metadata asset is obtained; The central theme of the table structure of each of the table metadata assets is matched with the subject words of each domain to obtain the sub-domain type to which each of the table metadata assets belongs.

6. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data asset completion method according to any one of claims 1 to 4 is implemented.

7. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the data asset completion method as described in any one of claims 1 to 4.

Images