Data catalog construction method, apparatus, medium, and device

Abstract

Embodiments of the present application disclose a data catalog construction method, device, medium and equipment, the data catalog construction method comprises: acquiring metadata in a plurality of data storage platforms respectively, reducing data transmission quantity, then performing entity recognition and entity relationship recognition on the metadata to extract entities in the metadata and the association relationship between the entities, constructing a global metadata static knowledge graph according to the entities and the association relationship, obtaining a knowledge graph with high accuracy and wide data coverage, eliminating the data island situation between various data storage platforms, then acquiring a directory construction strategy corresponding to the data catalog according to the type of the data catalog to be constructed, extracting directory information from the global metadata static knowledge graph according to the directory construction strategy, generating a data catalog according to the obtained directory information, and then facilitating data analysis and being conducive to data mining and utilization.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data catalog construction method, apparatus, computer-readable storage medium, and electronic device. Background Technology

[0002] Big data represents a new stage in the development of information technology. With the convergence and integration of information technology with human production and daily life, and the rapid popularization of the internet, global data is experiencing explosive growth and massive accumulation. In the wave of big data industrialization, data-centric information infrastructure will evolve from private big data clouds to public clouds, and even gradually move into a hybrid cloud stage.

[0003] Enterprise data is characterized by abundant data resources and complex data sources. For example, operator data includes business support system data, operation support system data, management support system data, deep packet inspection (DPI) data, fixed network DPI data, signaling data, etc.

[0004] In traditional enterprise big data management, data lake is a relatively mainstream centralized data management method at present. It can centralize all kinds of raw data of an enterprise and provide storage, processing, analysis and transmission. However, it has problems such as long data circulation cycle, low global correlation of data and low data security. Summary of the Invention

[0005] To address the aforementioned technical problems, embodiments of this application provide a data catalog construction method, apparatus, computer-readable storage medium, and electronic device to improve data management effectiveness.

[0006] According to one aspect of the embodiments of this application, a data catalog construction method is provided, the method comprising:

[0007] Metadata from multiple data storage platforms is acquired, and entity recognition and entity relationship recognition are performed on the metadata to extract entities and relationships between entities.

[0008] Construct a global metadata static knowledge graph based on entities and relationships;

[0009] Obtain the directory building strategy corresponding to the data directory based on the type of the data directory to be built;

[0010] Based on the directory construction strategy, directory information is extracted from the global metadata static knowledge graph to generate a data directory.

[0011] In some embodiments, entity identification and entity relationship identification are performed on the metadata to extract entities and the relationships between entities in the metadata, including:

[0012] Entity identification is performed on the metadata of each data storage platform to obtain the entity database corresponding to each data storage platform;

[0013] Relationship identification is performed on entities in each entity database to obtain the association relationships between entities in each entity database.

[0014] In some embodiments, entity identification is performed on the metadata in each data storage platform to obtain an entity database corresponding to each data storage platform, including:

[0015] Retrieve the data type of the metadata;

[0016] Entity recognition strategy matching is performed based on data type to obtain the entity recognition strategy corresponding to the metadata;

[0017] Entity identification is performed on the metadata according to the entity identification strategy to obtain the entity corresponding to the metadata;

[0018] Add the entity corresponding to the metadata to the entity database of the data storage platform corresponding to the metadata.

[0019] In some embodiments, a global metadata static knowledge graph is constructed based on entities and relationships, including:

[0020] Obtain the preset initial knowledge graph and perform entity alignment for each entity library to obtain the entities to be added and their relationships;

[0021] The initial knowledge graph is completed based on the entities to be added and their relationships, resulting in a global metadata static knowledge graph.

[0022] In some embodiments, the data directory to be constructed is of the type of static data resource catalog; the directory information is extracted from the global metadata static knowledge graph according to the directory construction strategy, and a data directory is generated based on the obtained directory information, including:

[0023] Obtain the keyword table and relationship table of the catalog entries corresponding to the static data resource catalog to be constructed;

[0024] Based on the keyword table of directory items, keyword matching is performed on the global metadata static knowledge graph to obtain the directory item information contained in the global metadata static knowledge graph;

[0025] The catalog item information is arranged according to the catalog item relationship table to generate a static data resource catalog.

[0026] In some embodiments, the data directory to be constructed is a static business resource directory; the directory information is extracted from the global metadata static knowledge graph according to the directory construction strategy, and a data directory is generated based on the obtained directory information, including:

[0027] Retrieve the node attributes of each node in the global metadata static knowledge graph;

[0028] Each node is clustered based on its attributes to obtain the business category to which each node belongs.

[0029] A static business resource catalog is generated based on the business categories contained in the global metadata static knowledge graph and the nodes contained in each business category.

[0030] In some embodiments, the data directory to be constructed is a dynamic resource retrieval directory; the directory information is extracted from the global metadata static knowledge graph according to the directory construction strategy, and a data directory is generated based on the obtained directory information, including:

[0031] Receive resource retrieval information input by the user;

[0032] Extract search keywords from resource retrieval information;

[0033] Based on the search keywords, key information is extracted from the global metadata static knowledge graph to obtain the search results corresponding to the resource retrieval information;

[0034] A dynamic resource search directory is generated based on the search results.

[0035] According to one aspect of the embodiments of this application, a data catalog construction apparatus is provided, the apparatus comprising:

[0036] The entity and relationship extraction module is configured to acquire metadata from multiple data storage platforms, perform entity recognition and entity relationship recognition on the metadata, and extract the entities and the relationships between entities in the metadata.

[0037] The knowledge graph construction module is configured to build a global metadata static knowledge graph based on entities and relationships.

[0038] The directory building strategy confirmation module is configured to obtain the directory building strategy corresponding to the data directory based on the type of the data directory to be built.

[0039] The data directory generation module is configured to extract directory information from the global metadata static knowledge graph according to the directory construction strategy, and generate a data directory based on the obtained directory information.

[0040] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the data directory construction method described above.

[0041] According to one aspect of the embodiments of this application, an electronic device is provided, including one or more processors; and a storage device for storing one or more programs, which, when executed by the electronic device, cause the electronic device to implement the data directory construction method described above.

[0042] In the technical solution provided by the embodiments of this application, the amount of data transmission is reduced by acquiring metadata from multiple data storage platforms. Then, entity recognition and entity relationship recognition are performed on the metadata to extract entities and their relationships. A global metadata static knowledge graph is constructed based on the entities and their relationships, resulting in a knowledge graph with high accuracy and broad data coverage. This eliminates the data silos between different data storage platforms. Then, the directory construction strategy corresponding to the data directory to be constructed is obtained according to the type of the data directory to be constructed. Directory information is extracted from the global metadata static knowledge graph according to the directory construction strategy to generate a data directory based on the obtained directory information. This facilitates data analysis and is beneficial for data mining and utilization.

[0043] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0045] Figure 1 This is a schematic diagram illustrating the application environment of the data catalog construction method, as an exemplary embodiment of this application;

[0046] Figure 2 This is a flowchart illustrating a data catalog construction method in an exemplary embodiment of this application;

[0047] Figure 3 This is a schematic diagram illustrating metadata acquisition as shown in an exemplary embodiment of this application;

[0048] Figure 4 This is a flowchart illustrating a data catalog construction method in another exemplary embodiment of this application;

[0049] Figure 5 This is a flowchart illustrating a data catalog construction method in another exemplary embodiment of this application;

[0050] Figure 6 This is a flowchart illustrating a data catalog construction method in another exemplary embodiment of this application;

[0051] Figure 7 This is a block diagram illustrating a data catalog construction apparatus according to an exemplary embodiment of this application;

[0052] Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application embodiments. Detailed Implementation

[0053] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments identical to those described in this application. Rather, they are merely examples of apparatuses and methods identical to some aspects of this application as detailed in the appended claims.

[0054] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented as application programs, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0055] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0056] It should be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0057] Optionally, in this embodiment, the data catalog construction method can be applied to, for example... Figure 1 In the environment shown. For example... Figure 1As shown, the implementation environment includes a data storage platform 110 and a server 120. Multiple data storage platforms 110 and servers 120 can be directly or indirectly connected through wired or wireless communication, which is not limited in this application.

[0058] The data storage platform 110 is used for data storage. The data storage platform 110 can be a smartphone, tablet, laptop, desktop computer, computer cluster, etc., but is not limited to these. The embodiments of this application do not limit the number and type of data storage platforms.

[0059] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0060] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats, including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0061] Optionally, server 120 may undertake the construction of the primary data directory, and terminal 110 may undertake the construction of the secondary data directory; or server 120 may undertake the construction of the secondary data directory, and terminal 110 may undertake the construction of the primary data directory; or server 120 or terminal 110 may each undertake the construction of the data directory independently, and this application does not limit this.

[0062] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data catalog construction method in an exemplary embodiment of this application. This data catalog construction method can be applied to... Figure 1 The implementation environment shown is specifically executed by server 120 within that implementation environment. It should be understood that this method can also be applied to other exemplary implementation environments and executed by devices in other implementation environments; this embodiment does not limit the implementation environment to which the method is applicable.

[0063] The following section will describe in detail the data directory construction method proposed in this application embodiment, taking the server as the specific execution entity.

[0064] like Figure 2 As shown, in an exemplary embodiment, the data catalog construction method includes at least steps S210 to S240, which are described in detail below:

[0065] Step S210: Obtain metadata from multiple data storage platforms, perform entity recognition and entity relationship recognition on the metadata, and extract the entities in the metadata and the relationships between entities.

[0066] It should be noted that metadata refers to data that describes data attributes, used to support functions such as indicating storage location, historical data, resource lookup, and file records. An entity can refer to an objectively existing and distinguishable thing, or it can refer to an abstract entity in a specific domain that requires official definition. Relationships refer to the relationships between entities.

[0067] Each data storage platform communicates with the server to send metadata corresponding to the data stored in each platform to the server. For example, Company A deploys data storage platforms in different regions, each used to store business data generated in its respective region. The server that performs data catalog construction can obtain metadata from all data storage platforms, or it can obtain metadata from some data storage platforms. The data storage platforms from which metadata is obtained can be selected based on the actual application, and this application does not limit this selection.

[0068] For example, the server could send a metadata retrieval request to the data storage platform, causing the data storage platform to respond with metadata based on the request. See, for example, [link to relevant documentation]. Figure 3 , Figure 3 A diagram illustrating the acquisition of metadata, such as Figure 3 As shown, the user terminal communicates with the server. The user terminal sends a data catalog construction request to the server. This data catalog construction request carries relevant information about the data storage platform. The server identifies the data storage platforms that need to obtain metadata based on the relevant information of the data storage platforms, and sends metadata acquisition requests to these data storage platforms so that the data storage platforms can provide feedback on the metadata based on the metadata acquisition requests.

[0069] For example, the data storage platform can periodically upload its corresponding metadata to the server, such as uploading metadata to the server every day. In this case, the data storage platform can avoid bandwidth waste caused by data transmission by only uploading metadata change information to the server.

[0070] After obtaining the metadata of the data storage platform, identify the entities contained in the metadata and extract the relationships between the entities.

[0071] Optionally, a predefined entity classification system can be used, and then a corresponding algorithm can be used to automatically classify entities; alternatively, entity objects can be identified from metadata based on the semantic features of entities, and then clustering algorithms can be used to cluster the identified entity objects to obtain the corresponding entities; alternatively, metadata can be input into a pre-trained entity recognition model, which extracts features from the metadata to obtain word embedding vectors, and then performs entity recognition on the word embedding vectors to obtain the entities in the metadata. The entity recognition model can be a BERT (Bidirectional Encoder Representations from Transformers) model, a Long Short-Term Memory (LSTM) neural network model, a Recurrent Neural Network (RNN) model, or other neural network models, which are not limited in this application.

[0072] Furthermore, after entities are identified through entity recognition, relationships between these entities are extracted. For example, a relationship value between a first entity and a second entity is calculated. This relationship value is the probability value of the first entity and the second entity appearing simultaneously. The relationship value is compared with a preset relationship value threshold. The relationship between entities whose relationship value is not less than the preset relationship value threshold is determined as the relationship between the first entity and the second entity.

[0073] Step S220: Construct a global metadata static knowledge graph based on entities and relationships.

[0074] It should be noted that a knowledge graph refers to a knowledge graph composed of (entity-relationship-entity) triples, which describes entities and the relationships between entities in a visual way.

[0075] Obtaining the relationships between entities yields multiple triples, which are then used to construct a global metadata static knowledge graph. This process can be understood as connecting multiple triples into a semantic network based on the relationships between elements within the merged triples. Each node in the semantic network corresponds to an entity type or attribute within a triple, and the relationships between nodes correspond to the association information within the triples.

[0076] Step S230: Obtain the directory construction strategy corresponding to the data directory based on the type of the data directory to be constructed.

[0077] It should be noted that the data directory is used to index the data. The directory construction strategy is used to define the creation rules when creating the data directory based on the global metadata static knowledge graph, such as the directory items in the pre-created data directory and the relationships between the various directory items.

[0078] It is understandable that different types of data directories require different directory building strategies, and the type of data directory to be built can be determined based on the data directory requirements input by the user.

[0079] Optionally, the server may pre-store a directory type and build strategy mapping table. This mapping table records the directory build strategies corresponding to each type of data directory. The directory type and build strategy mapping table can be queried by the type of the data directory to be built to confirm the directory build strategy corresponding to the data directory to be built.

[0080] Step S240: Extract directory information from the global metadata static knowledge graph according to the directory construction strategy, and generate a data directory based on the obtained directory information.

[0081] It should be noted that the directory information refers to the information corresponding to each directory item in the data directory to be constructed, which is the specific content that makes up the data directory.

[0082] Based on the directory construction strategy, directory information is extracted from the global metadata static knowledge graph to obtain the directory item information corresponding to the data directory to be constructed. Then, the data directory is generated according to the relationship between the information of each directory item. The relationship between the information of each directory item includes, but is not limited to, the hierarchical relationship between directory items and the order relationship between directory items, etc., which are not limited in this application.

[0083] In related technologies, due to the distributed storage characteristics of enterprise data, enterprises will set up multiple data storage platforms. Therefore, when performing aggregation and calculation on the data from these data storage platforms, data aggregation and transmission are required, resulting in a large amount of data transmission in the transmission network, which can easily cause network congestion and waste of resources. Moreover, these data storage platforms are relatively scattered, making it difficult to apply global data and resulting in poor data value mining effects.

[0084] The data catalog construction method provided in this application obtains metadata from multiple data storage platforms, reducing the amount of data transmission. Then, it performs entity recognition and entity relationship recognition on the metadata to extract entities and their relationships. Based on the entities and relationships, it constructs a global metadata static knowledge graph, resulting in a knowledge graph with high accuracy and broad data coverage, eliminating data silos between different data storage platforms. Then, it obtains the corresponding catalog construction strategy based on the type of data catalog to be constructed, and extracts catalog information from the global metadata static knowledge graph according to the catalog construction strategy to generate a data catalog. This facilitates data analysis and is beneficial for data mining and utilization.

[0085] In some implementations, entity identification and entity relationship identification are performed on the metadata to extract entities in the metadata and the relationships between entities, including: performing entity identification on the metadata in each data storage platform to obtain the entity database corresponding to each data storage platform; and performing relationship identification on the entities in each entity database to obtain the relationships between entities in each entity database.

[0086] To improve the accuracy of entity and entity relationship extraction, entity identification is performed on the metadata of each data storage platform to obtain the entity database corresponding to each data storage platform. Then, the relationship between entities in each entity database is identified to obtain the relationship between entities in each entity database.

[0087] In some implementations, entity identification is performed on the metadata in each data storage platform to obtain the entity library corresponding to each data storage platform. This includes: obtaining the data type of the metadata; matching the entity identification strategy according to the data type to obtain the entity identification strategy corresponding to the metadata; performing entity identification on the metadata according to the entity identification strategy to obtain the entity corresponding to the metadata; and adding the entity corresponding to the metadata to the entity library of the data storage platform corresponding to the metadata.

[0088] Optionally, metadata data types include structured data, semi-structured data, and unstructured data. Structured data refers to data managed in the form of relational database tables, whose data storage and arrangement are regular; semi-structured data refers to data with a basic fixed structure pattern, such as log files, XML documents, JSON documents, and emails; unstructured data refers to data without a fixed pattern, such as WORD, PDF, PPT, EXL, and various formats of images and videos.

[0089] Since different data types have different representations, it is necessary to match the corresponding entity recognition strategy according to the data type of the metadata. This allows for entity recognition of each type of metadata according to the entity recognition strategy, thereby improving the accuracy of entity recognition. Furthermore, since entities can be directly identified from structured data, by distinguishing structured data and performing entity extraction and other calculations only on semi-structured and unstructured data, computational resources can be saved and entity recognition efficiency can be improved.

[0090] For example, the server stores a semi-structured and unstructured data entity recognition model (hereinafter referred to as the entity recognition model). Metadata belonging to the semi-structured and unstructured data is input into the entity recognition model, and the entity recognition model extracts entities from the semi-structured and unstructured data. For example, the entity recognition model extracts features from the input text data to obtain the embedded vector corresponding to the text data, and then the entity recognition model extracts the entities contained in the text data based on the embedded vector.

[0091] Before inputting semi-structured and unstructured data into the entity recognition model, the process also includes preprocessing the semi-structured and unstructured data.

[0092] Since the encoding formats of semi-structured and unstructured data may not be uniform, and semi-structured and unstructured data may contain web page tags, this application embodiment can perform preprocessing such as transcoding and tag removal on the semi-structured and unstructured data to obtain the corresponding text data. After obtaining the text data corresponding to the semi-structured and unstructured data, word segmentation can be performed on the text data, and then low-frequency words in the word segmentation results can be removed to obtain a candidate word set. The candidate word set is then input into the entity recognition model to obtain the entity output by the entity recognition model.

[0093] Furthermore, a global metadata static knowledge graph is constructed based on entities and relationships, including: obtaining a preset initial knowledge graph and aligning each entity library to obtain the entities to be added and their relationships; and completing the initial knowledge graph based on the entities to be added and their relationships to obtain the global metadata static knowledge graph.

[0094] The initial knowledge graph can be a knowledge graph manually annotated based on expert experience and rules. Based on this initial knowledge graph, incremental updates are performed to obtain a global metadata static knowledge graph.

[0095] Entity alignment between each entity library includes entity disambiguation and referential resolution.

[0096] Entity disambiguation refers to calculating the specific meaning of an entity, while referential disambiguation refers to calculating the specific content referred to by each referential term.

[0097] Entity disambiguation aligns data from different sources into a single entity. For example, the entity "Cheng Mou" might be referred to as "Cheng Mou's Big Brother" or "Mou Chan" in other data sources. The Word2vec algorithm can be used for entity disambiguation. For instance, given two entity vectors Xword = (x1, x2, x3, ..., xn) and Yword = (y1, y2, y3, ..., yn), the distance between Xword and Yword is calculated. Distance calculation methods include, but are not limited to, Euclidean distance and cosine distance. The similarity between the two entities is obtained based on the distance between Xword and Yword. When the similarity exceeds a set similarity threshold, the two entities can be considered to refer to the same meaning, thus achieving entity disambiguation.

[0098] Reference resolution can effectively solve the problem of ambiguous reference in text. It can obtain the contextual information of the entity or relationship to be resolved, extract keywords from the context, and then replace the entity to be resolved with these keywords in the original text. Semantic features of the replaced text are extracted, and the probability of the entity or relationship to be resolved referring to the keyword is calculated based on the semantic features. The keyword with the highest probability is selected as the content referred to by the entity or relationship to be resolved.

[0099] Then, by aligning entities in each entity library, the entities to be incremented and their relationships are obtained, which are then used to complete the initial knowledge graph. For example, the initial knowledge graph and the entities and relationships to be incremented are fused to obtain an intermediate knowledge graph. This intermediate knowledge graph is then input into a pre-trained knowledge graph completion model to further predict and mine possible relationships and entities in the intermediate knowledge graph. Finally, the global metadata static knowledge graph output by the knowledge graph completion model is obtained.

[0100] This global metadata static knowledge graph connects isolated data due to geographical location, business type, and other reasons, avoiding data silos and facilitating the creation of subsequent data directories.

[0101] Please see Figure 4 , Figure 4 This is a flowchart illustrating a data catalog creation method according to another exemplary embodiment. For example... Figure 4 As shown, in an exemplary embodiment, the type of the data directory to be constructed is static data resource cataloging; step S240 involves extracting directory information from the global metadata static knowledge graph according to the directory construction strategy, in order to generate a data directory based on the obtained directory information, including:

[0102] Step S2411: Obtain the directory item keyword table and directory item relationship table corresponding to the static data resource catalog to be constructed;

[0103] Step S2412: Perform keyword matching on the global metadata static knowledge graph based on the directory item keyword table to obtain the directory item information contained in the global metadata static knowledge graph;

[0104] Step S2413: Arrange the catalog item information according to the catalog item relationship table to generate a static data resource catalog.

[0105] It should be noted that static data resource cataloging refers to analyzing, selecting, describing, and recording each entity in the global metadata static knowledge graph according to preset standards and rules, and then organizing the entries into a catalog in a certain order.

[0106] The directory item keyword table contains directory item information that needs to be generated in the static data resource cataloging to be constructed. For example, the directory item keywords in the directory item keyword table include "user internet access data", "user account data", "SMS data", etc. The directory item relationship table refers to the relationships between various directory item information. For example, the directory item relationships in the directory item relationship table include "parallel relationship" and "inclusion relationship". For example, if user internet access data includes user SMS data, then the directory item information corresponding to the directory item keyword "user internet access data" and the directory item information corresponding to the directory item keyword "SMS data" have an inclusion relationship.

[0107] Based on the keyword list, keyword matching is performed on the global metadata static knowledge graph to obtain the directory item information contained in the global metadata static knowledge graph. Then, the directory item information is arranged according to the directory item relationship table to generate a static data resource catalog.

[0108] Using the keyword table and relationship table of catalog items as cataloging control value fields, the catalogs of different categories and levels of data resources on various data storage platforms are reorganized and sorted to form static data resource catalogs, which can meet the management, discovery, location and sharing of data resources from the perspective of resource classification.

[0109] Please see Figure 5 , Figure 5 This is a flowchart illustrating a data catalog creation method according to another exemplary embodiment. For example... Figure 5 As shown, in an exemplary embodiment, the data directory to be constructed is a static business resource directory; step S240 involves extracting directory information from the global metadata static knowledge graph according to the directory construction strategy, and generating a data directory based on the obtained directory information, including:

[0110] Step S2421: Obtain the node attributes of each node in the global metadata static knowledge graph;

[0111] Step S2422: Cluster each node according to its attributes to obtain the business category to which each node belongs;

[0112] Step S2423: Generate a static business resource catalog based on the business categories contained in the global metadata static knowledge graph and the nodes contained in each business category.

[0113] A global metadata static knowledge graph example has at least two nodes, each with a corresponding node attribute value. The node attribute value can be specific content, or it can be "default" or "empty". This application is not limited by the number of node attribute values ​​or the specific content of the information nodes in the graph element.

[0114] For example, clustering each node based on its attributes specifically includes:

[0115] 2.1 All nodes in the global metadata static knowledge graph are taken as nodes to be processed, resulting in a set of nodes to be processed.

[0116] 2.2 Take one node from the set of nodes to be processed in turn, such as node i.

[0117] 2.3 Get the node attributes of node i.

[0118] 2.4 Determine the business category corresponding to node i.

[0119] If node i is the first node to be processed, meaning the current number of business categories is zero, then a new business category is determined based on the attributes of node i. The description information of this business category is the node attributes of node i.

[0120] If node i is not the first node to be processed, it means that there is at least one business category at present, then:

[0121] 1) Calculate the similarity between the node attributes of node i and the description information of each existing business category to obtain the node-category similarity value.

[0122] This can involve extracting features from the node attributes of node i and the description information of each business category, and then calculating the similarity between the two extracted feature vectors. The method for calculating the similarity can be the calculation of the Euclidean distance, cosine distance, etc. between the two feature vectors, and this application does not limit this.

[0123] 2) If the similarity value between a node and its category is greater than the average similarity value between all nodes in the corresponding business category and the description information of that business category, then node i is classified into that business category.

[0124] If multiple business categories meet the condition that the node and category similarity values ​​are greater than the mean, then determine the difference between the node and category similarity values ​​and each mean, select the business category with the largest difference, and classify node i into that business category.

[0125] 3) If the similarity value between a node and its category is not greater than the average similarity value between all nodes in all business categories and the description information of that business category, then a new business category is determined based on the node attributes of node i, and the description information of that business category is the node attributes of node i.

[0126] 2.5 Remove node i from the set of nodes to be processed.

[0127] 2.6 Determine all nodes that have edges with node i through the global metadata static knowledge graph. All nodes must be in the set of nodes to be processed. If they are not in the set, it means that they have already been processed and the node is ignored.

[0128] For each node with an edge, such as node j:

[0129] 1) Calculate the edge node and node similarity value xij between the node attributes of node j and the node attributes of node i.

[0130] 2) Calculate the similarity between the node attributes of node j and the description information of each business category k to obtain the edge node-type similarity value x. kj And obtain the node and type similarity value x between the node attributes of node i and the description information of each business category k. ki .

[0131] 3) The final similarity value between node j and each business category is determined as: x j =1+α.

[0132] Where α is the adjustment coefficient, α = x ij x kj x ki .

[0133] 4) If x j If the similarity value between node j and the description information of the business category is greater than the average value of all nodes in the business category, then node j will be classified into the business category and removed from the set of nodes to be processed.

[0134] If multiple business categories satisfy x j If the value is greater than the average, then node j is classified into the business category to which node i belongs.

[0135] 5) If x j If the similarity value between node j and the description information of the business category is not greater than the average value of all nodes in all business categories, then node j is assigned to the business category to which node i belongs, and node j is deleted from the set of nodes to be processed.

[0136] Repeat steps 2.1 to 2.6 until the set of nodes to be processed is empty. This forms the final clustering result, which is then used to generate a static business resource catalog.

[0137] For example, the basic business data in the generated static business resource catalog can be divided into fixed communication business data, cellular mobile communication business data, satellite communication business data, Internet business data, Internet Protocol (IP) telephony call data, trunking communication business data, wireless paging business data, etc. It is understood that these basic business data can be further subdivided into sub-business catalogs according to each category, and this application does not limit this.

[0138] Through the above process, data can be segmented according to business domains, and corresponding data can be selected from the global static metadata knowledge graph according to business needs. This data is then centrally stored and processed in a hierarchical business format to form a static business resource catalog.

[0139] Please see Figure 6 , Figure 6 This is a flowchart illustrating a data catalog creation method according to another exemplary embodiment. For example... Figure 6 As shown, in an exemplary embodiment, the type of the data directory to be constructed is a dynamic resource retrieval directory; step S240 involves extracting directory information from the global metadata static knowledge graph according to the directory construction strategy, in order to generate a data directory based on the obtained directory information, including:

[0140] Step S2431: Receive resource retrieval information input by the user;

[0141] Step S2432: Extract search keywords from resource retrieval information;

[0142] Step S2433: Extract key information from the global metadata static knowledge graph based on the search keywords to obtain the search results corresponding to the resource search information;

[0143] Step S2434: Generate a dynamic resource search catalog based on the search results.

[0144] The system obtains the search keywords corresponding to the resource retrieval information input by the user, extracts key information from the global metadata static knowledge graph based on the search keywords, and then generates a dynamic resource retrieval directory based on the extracted search results.

[0145] For example, if the search keyword obtained from the resource retrieval information is "telephone call data", then the key information is extracted from the global metadata static knowledge graph based on the search keyword. The search results corresponding to the resource retrieval information contain all telephone call data in each data storage platform corresponding to the global metadata static knowledge graph.

[0146] The dynamic resource search directory is automatically generated based on user-input requirements. Its characteristics include real-time performance, cross-domain compatibility, and intelligence. It can dynamically cover multiple data storage platforms for collaborative indexing. Furthermore, it can perform semantic recognition based on user queries to extract key features from information to generate the dynamic resource search directory.

[0147] This dynamic resource retrieval catalog can be a subgraph of a global metadata static knowledge graph, or a new knowledge graph obtained by fusing multiple subgraphs from multiple data storage platforms. It can cover all the metadata information that users need, helping users achieve full coverage of their target data.

[0148] It is understood that this application can generate one or more of the following based on the global metadata static knowledge graph: static data resource catalog, static business resource directory, and dynamic resource retrieval directory. It can also generate other types of data directories based on the global metadata static knowledge graph. This application does not limit this.

[0149] The data catalog construction method provided in this application acquires metadata from multiple data storage platforms, reducing data transmission volume. Then, it performs entity recognition and entity relationship recognition on the metadata to extract entities and their relationships. Based on these entities and relationships, a global metadata static knowledge graph is constructed, resulting in a highly accurate and comprehensive knowledge graph that eliminates data silos between different data storage platforms. Next, based on the type of data catalog to be constructed, a corresponding catalog construction strategy is obtained. Based on this strategy, catalog information is extracted from the global metadata static knowledge graph to generate a data catalog, facilitating data analysis and enabling data mining and utilization.

[0150] Figure 7 This is a block diagram illustrating a data catalog construction apparatus according to an embodiment of this application, such as Figure 7 As shown, the device includes:

[0151] The entity and relationship extraction module 710 is configured to acquire metadata from multiple data storage platforms, perform entity recognition and entity relationship recognition on the metadata, and extract the entities and the relationships between entities in the metadata.

[0152] Knowledge graph construction module 720 is configured to build a global metadata static knowledge graph based on entities and relationships;

[0153] The directory building strategy confirmation module 730 is configured to obtain the directory building strategy corresponding to the data directory based on the type of the data directory to be built.

[0154] The data catalog generation module 740 is configured to extract catalog information from the global metadata static knowledge graph according to the catalog building strategy, so as to generate a data catalog based on the obtained catalog information.

[0155] In one embodiment of this application, the entity and relation extraction module 710 may include:

[0156] The entity extraction unit is configured to perform entity identification on the metadata of each data storage platform to obtain the entity database corresponding to each data storage platform.

[0157] The relationship extraction unit is configured to identify the relationships between entities in each entity database and obtain the association relationships between entities in each entity database.

[0158] In one embodiment of this application, the entity extraction unit may include:

[0159] The data type retrieval unit is configured to retrieve the data type of metadata.

[0160] The entity recognition strategy matching unit is configured to perform entity recognition strategy matching based on data type to obtain the entity recognition strategy corresponding to the metadata.

[0161] The entity recognition unit is configured to perform entity recognition on metadata according to an entity recognition strategy to obtain the entity corresponding to the metadata.

[0162] The entity addition unit is configured to add the entity corresponding to the metadata to the entity library of the data storage platform corresponding to the metadata.

[0163] In one embodiment of this application, the knowledge graph construction module 720 may include:

[0164] The entity alignment unit is configured to obtain a preset initial knowledge graph and perform entity alignment on each entity library to obtain the entities to be added and their relationships.

[0165] The graph completion unit is configured to complete the initial knowledge graph based on the entities to be added and their relationships, thereby obtaining a global metadata static knowledge graph.

[0166] In one embodiment of this application, the data catalog to be constructed is of the type of static data resource catalog; the data catalog generation module 740 may include:

[0167] The standard acquisition unit is configured to acquire the directory item keyword table and directory item relationship table corresponding to the static data resource catalog to be constructed;

[0168] The directory item information extraction unit is configured to perform keyword matching on the global metadata static knowledge graph based on the directory item keyword table to obtain the directory item information contained in the global metadata static knowledge graph;

[0169] The first directory generation unit is configured to arrange the directory item information according to the directory item relationship table and generate a static data resource catalog.

[0170] In one embodiment of this application, the data directory to be constructed is a static business resource directory; the data directory generation module 740 may further include:

[0171] The node attribute acquisition unit is configured to acquire the node attributes of each node in the global metadata static knowledge graph;

[0172] The clustering processing unit is configured to perform clustering processing on each node based on node attributes to obtain the business category to which each node belongs;

[0173] The second directory generation unit is configured to generate a static business resource directory based on the business categories contained in the global metadata static knowledge graph and the nodes contained in each business category.

[0174] In one embodiment of this application, the data directory to be constructed is a dynamic resource retrieval directory; the data directory generation module 740 may further include:

[0175] The resource retrieval information receiving unit is configured to receive resource retrieval information input by the user.

[0176] The keyword extraction unit is configured to extract search keywords from resource retrieval information.

[0177] The search result acquisition unit is configured to extract key information from the global metadata static knowledge graph based on the search keywords, and obtain the search results corresponding to the resource search information.

[0178] The third directory generation unit is configured to generate a dynamic resource retrieval directory based on the search results.

[0179] It should be noted that the data directory construction apparatus and the data directory construction method provided in the above embodiments belong to the same concept. The specific way each module and unit performs operations has been described in detail in the method embodiments, and will not be repeated here. In practical applications, the data directory construction apparatus provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation here.

[0180] Figure 8A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0181] It should be noted that, Figure 8 The computer system 800 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0182] like Figure 8 As shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processing unit 810, at least one storage unit 820, a bus 830 connecting different system components (including storage unit 820 and processing unit 810), and a display unit 840.

[0183] The storage unit stores program code, which can be executed by the processing unit 810, causing the processing unit 810 to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this disclosure.

[0184] Storage unit 820 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 821 and / or cache memory 822, and may further include a read-only memory (ROM) 823.

[0185] The storage unit 820 may also include a program / utility 824 having a set (at least one) of program modules 825, including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0186] Bus 830 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0187] Electronic device 800 can also communicate with one or more external devices 870 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or application modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0188] In particular, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer applications. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. When the computer program is executed by the processing unit 810, it performs various functions defined in the system of this application.

[0189] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can initiate, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0190] The units described in the embodiments of this application can be implemented by application programs or by hardware, and the described units can also be located in a processor. The names of these units do not necessarily constitute a limitation on the unit itself.

[0191] Another aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned data directory construction method. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0192] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data directory construction method provided in each of the above embodiments.

[0193] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. A method for constructing a data directory, characterized in that, include: Metadata from multiple data storage platforms is acquired, and entity recognition and entity relationship recognition are performed on the metadata to extract the entities in the metadata and the relationships between the entities; Construct a global metadata static knowledge graph based on the entities and the relationships; Obtain the directory construction strategy corresponding to the data directory based on the type of the data directory to be constructed; Based on the directory construction strategy, the directory information of the global metadata static knowledge graph is extracted to generate a data directory. The data directory to be constructed is of the type of static data resource catalog; The step of extracting directory information from the global metadata static knowledge graph according to the directory construction strategy, and generating a data directory based on the obtained directory information, includes: obtaining a directory item keyword table and a directory item relationship table corresponding to the static data resource catalog to be constructed; The global metadata static knowledge graph is matched with keywords based on the directory item keyword table to obtain the directory item information contained in the global metadata static knowledge graph; the directory item information is arranged according to the directory item relationship table to generate the static data resource catalog.

2. The method according to claim 1, characterized in that, The step of performing entity identification and entity relationship identification on the metadata to extract entities in the metadata and the relationships between those entities includes: Entity identification is performed on the metadata in each of the data storage platforms to obtain the entity database corresponding to each data storage platform; Relationship identification is performed on each entity in the entity database to obtain the association relationships between entities in each entity database.

3. The method according to claim 2, characterized in that, The step of performing entity identification on the metadata in each of the data storage platforms to obtain the entity database corresponding to each data storage platform includes: Obtain the data type of the metadata; Entity recognition strategy matching is performed based on the data type to obtain the entity recognition strategy corresponding to the metadata; The entity recognition strategy is used to perform entity recognition on the metadata to obtain the entity corresponding to the metadata; Add the entity corresponding to the metadata to the entity library of the data storage platform corresponding to the metadata.

4. The method according to claim 2, characterized in that, The step of constructing a global metadata static knowledge graph based on the entities and the relationships includes: Obtain a preset initial knowledge graph and perform entity alignment on each entity library to obtain the entities to be incrementalized and their relationships; Based on the entities to be added and their relationships, the initial knowledge graph is completed to obtain a global metadata static knowledge graph.

5. The method according to any one of claims 1 to 4, characterized in that, The data directory to be constructed is of the type of static business resource directory; the step of extracting directory information from the global metadata static knowledge graph according to the directory construction strategy, and generating a data directory based on the obtained directory information, includes: Obtain the node attributes of each node in the global metadata static knowledge graph; Based on the node attributes, each node is clustered to obtain the business category to which each node belongs; The static business resource directory is generated based on the business categories contained in the global metadata static knowledge graph and the nodes contained in each business category.

6. The method according to any one of claims 1 to 4, characterized in that, The data directory to be constructed is a dynamic resource retrieval directory; the step of extracting directory information from the global metadata static knowledge graph according to the directory construction strategy, and generating a data directory based on the obtained directory information, includes: Receive resource retrieval information input by the user; Extract search keywords from the resource retrieval information; Based on the search keywords, key information is extracted from the global metadata static knowledge graph to obtain the search results corresponding to the resource search information; The dynamic resource search catalog is generated based on the search results.

7. A data catalog construction apparatus, characterized in that, include: The entity and relationship extraction module is configured to acquire metadata from multiple data storage platforms, perform entity identification and entity relationship identification on the metadata, and extract the entities in the metadata and the relationships between the entities. The knowledge graph construction module is configured to construct a global metadata static knowledge graph based on the entities and the relationships. The directory building strategy confirmation module is configured to obtain the directory building strategy corresponding to the data directory based on the type of the data directory to be built. The data directory generation module is configured to extract directory information from the global metadata static knowledge graph according to the directory construction strategy, so as to generate a data directory based on the obtained directory information; The data directory to be constructed is of the type of static data resource catalog; The step of extracting directory information from the global metadata static knowledge graph according to the directory construction strategy, and generating a data directory based on the obtained directory information, includes: obtaining a directory item keyword table and a directory item relationship table corresponding to the static data resource catalog to be constructed; The global metadata static knowledge graph is matched with keywords based on the directory item keyword table to obtain the directory item information contained in the global metadata static knowledge graph; the directory item information is arranged according to the directory item relationship table to generate the static data resource catalog.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data directory construction method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: processor; as well as A memory for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the data directory construction method as described in any one of claims 1 to 6.

Images