Data weaving processing method, device, electronic device and readable storage medium

Abstract

The present disclosure provides a data weaving processing method, device, electronic device and readable storage medium, which belongs to the field of communication technology. The method collects and stores metadata for the original data distributedly stored in the data source node, without the need to centrally transmit the data of a large number of edge nodes to the data center, reducing the cost of data replication, transmission and storage, improving data governance efficiency, reducing network congestion, bandwidth costs, and data center computing power requirements, and avoiding possible security and privacy risks during data replication, transmission and storage, improving overall data processing, access and management; by collecting metadata of the original data and building a knowledge graph based on metadata, data integration across businesses and systems between data source nodes is realized, and the degree of global data correlation is improved; and a dynamic data knowledge catalog is generated for users' real-time data retrieval needs, providing a unified, comprehensive and intuitive data view, and realizing accurate retrieval and access to data.

Description

Technical Field

[0001] The present disclosure belongs to the field of communication technology, and specifically relates to a data weaving processing method, device, electronic device and readable storage medium. Background Art

[0002] With the continuous development of network communications and the continuous advancement of informatization, data has shown explosive growth and massive aggregation in the process of information technology, production and life, which has brought new challenges to data governance.

[0003] At present, the method of entering data into a data lake is often used to achieve modern data governance, centralizing scattered, independent, and different types of original data to provide corresponding storage, management, transmission, analysis and other data processing services.

[0004] However, on the one hand, the above solution requires that the data generated and stored at the edge be transmitted to the data center in large quantities, which makes it difficult to support real-time services, easily blocks the network, and has high bandwidth costs. In addition, it is necessary to concentrate computing power in the data center to aggregate and calculate the returned data, which requires high computing power. As a result, there are problems such as long data flow cycle, high computing power performance bottleneck, and inflexible business processing. On the other hand, concentrating the original data of each edge in different data centers causes the data to be dispersed in different collection and computing platform clusters, making it impossible to provide globally aggregated data services, resulting in a low degree of global data correlation. On the other hand, in the process of centralized storage, management, transmission, and analysis, data security and privacy are difficult to fully guarantee, resulting in the inability to fully utilize the data. Summary of the invention

[0005] The purpose of the embodiments of the present disclosure is to provide a data weaving processing method, device, electronic device and readable storage medium, which can solve the problems of long data flow cycle, high computing power performance bottleneck, low global correlation and insufficient utilization in data governance solutions.

[0006] In order to solve the above technical problems, the present disclosure is implemented as follows:

[0007] In a first aspect, the present disclosure provides a data weaving processing method, which may include: collecting and storing metadata corresponding to original data, the original data is distributedly stored in more than two data source nodes, and the metadata is used to describe the storage information of the original data in the distributed storage; extracting entities and the relationships between entities in the metadata, and constructing a knowledge graph corresponding to the metadata based on the entities and relationships; determining the user's data retrieval needs; generating a dynamic data knowledge catalog in the knowledge graph based on the data retrieval needs, and the dynamic data knowledge catalog includes target metadata determined in the metadata based on the data retrieval needs.

[0008] Optionally, entities and relationships between entities are extracted from the metadata, and a knowledge graph corresponding to the metadata is constructed based on the entities and relationships, including: dividing the metadata into first metadata and second metadata; identifying a first entity in the first metadata, and extracting a first relationship between the first entities in the first metadata; constructing a first knowledge representation corresponding to the first metadata based on the first entity and the first relationship; and updating the first knowledge representation based on the second metadata to obtain a knowledge graph corresponding to the metadata.

[0009] Optionally, the first knowledge representation is updated according to the second metadata to obtain a knowledge graph corresponding to the metadata, including: identifying second entities in the second metadata, and extracting second relationships between second entities in the second metadata; performing entity alignment on the second entities, wherein the entity alignment includes one or more of entity disambiguation and coreference resolution; adding the second entity and the second relationship to the first knowledge representation to obtain a second knowledge representation; and using a knowledge representation learning model in the second knowledge representation to perform entity prediction and relationship prediction to obtain a knowledge graph.

[0010] Optionally, after extracting entities and relationships between entities in the metadata and constructing a knowledge graph corresponding to the metadata based on the entities and relationships, it also includes: associating the knowledge graph with the original data based on the corresponding relationship between the metadata and the original data; and constructing a static data knowledge directory corresponding to the metadata based on the knowledge graph.

[0011] Optionally, a dynamic data knowledge catalog is generated in the knowledge graph according to data retrieval requirements. The dynamic data knowledge catalog includes, after the target metadata determined in the metadata according to the data retrieval requirements, also including: obtaining associated metadata corresponding to the target metadata according to preset association relationships, the preset association relationships including one or more of causal association relationships, correlation association relationships, and integrated association relationships; and providing a data map to the user based on the target metadata and associated metadata.

[0012] Optionally, after collecting and storing metadata corresponding to the original data, it also includes: classifying and managing the metadata, and the classification management includes one or more of metadata statistics, metadata model definition, metadata classification rule definition, metadata version management, and metadata change management.

[0013] Optionally, after generating a dynamic data knowledge catalog in the knowledge graph according to data retrieval requirements, it also includes: obtaining corresponding target data in the original data according to the target metadata, and the data source node corresponding to the target data; obtaining the corresponding target algorithm according to the data retrieval requirements and the target data; decomposing the target algorithm into the data source nodes corresponding to the target data, and performing joint training between the data source nodes based on federated learning to parse the target data; and providing the parsing results of the target data based on the target algorithm.

[0014] In the second aspect, the present disclosure also provides a data weaving processing device, which may include an acquisition and storage module, a knowledge graph module, a user demand module, and a data directory module; wherein the acquisition and storage module is used to acquire and store metadata corresponding to the original data, and the original data is distributedly stored in more than two data source nodes, and the metadata is used to describe the storage information of the original data in the distributed storage; the knowledge graph module is used to extract entities and the relationships between entities in the metadata, and construct a knowledge graph corresponding to the metadata based on the entities and relationships; the user demand module is used to determine the user's data retrieval needs; the data directory module is used to generate a dynamic data knowledge directory in the knowledge graph according to the data retrieval needs, and the first knowledge directory includes the target metadata determined in the metadata according to the data retrieval needs.

[0015] Optionally, the knowledge graph module includes a data partitioning submodule, a first metadata processing submodule, a first knowledge representation construction submodule and a knowledge graph updating submodule; wherein the data partitioning submodule is used to divide the metadata into first metadata and second metadata; the first metadata processing submodule is used to identify the first entity in the first metadata and extract the first relationship between the first entities in the first metadata; the first knowledge representation construction submodule is used to construct the first knowledge representation corresponding to the first metadata based on the first entity and the first relationship; the knowledge graph updating submodule is used to update the first knowledge representation according to the second metadata to obtain the knowledge graph corresponding to the metadata.

[0016] Optionally, the knowledge graph update submodule includes a second metadata processing unit, an entity alignment unit, a second knowledge representation construction unit and a knowledge graph prediction update unit; wherein the second metadata processing unit is used to identify the second entity in the second metadata and extract the second relationship between the second entities in the second metadata; the entity alignment unit is used to perform entity alignment on the second entity, and the entity alignment includes one or more of entity disambiguation and reference resolution; the second knowledge representation construction unit is used to add the second entity and the second relationship to the first knowledge representation to obtain the second knowledge representation; the knowledge graph prediction update unit is used to use the knowledge representation learning model in the second knowledge representation to perform entity prediction and relationship prediction to obtain a knowledge graph.

[0017] Optionally, the data directory module is also used to associate the knowledge graph with the original data based on the corresponding relationship between the metadata and the original data, and to construct a static data knowledge directory corresponding to the metadata based on the knowledge graph.

[0018] Optionally, the user demand module is also used to obtain associated metadata corresponding to the target metadata based on preset association relationships, and provide a data map to the user based on the target metadata and associated metadata. The preset association relationships include one or more of causal association relationships, correlation association relationships, and integrated association relationships.

[0019] Optionally, the device may also include a data management module; wherein the data management module is used to classify and manage metadata, and the classification management includes one or more of metadata statistics, metadata model definition, metadata classification rule definition, metadata version management, and metadata change management.

[0020] Optionally, the device may also include a model application module; wherein the model application module is used to obtain corresponding target data and a data source node corresponding to the target data in the original data according to the target metadata, and obtain a corresponding target algorithm according to data retrieval requirements and the target data, and then decompose the target algorithm into the data source node corresponding to the target data, perform joint training between the data source nodes based on federated learning to parse the target data, and provide parsing results of the target data based on the target algorithm.

[0021] In a third aspect, the present disclosure provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the data weaving processing method of the first aspect.

[0022] In a fourth aspect, the present disclosure provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the data weaving processing method of the first aspect are implemented.

[0023] In a fifth aspect, the present disclosure provides a chip, comprising a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run a program or instruction to implement the steps of the data weaving processing method of the first aspect.

[0024] In a sixth aspect, the present disclosure provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the steps of implementing the data weaving processing method of the first aspect.

[0025] In the data weaving processing method provided by the present disclosure, metadata corresponding to the original data can be collected, wherein the original data is distributedly stored in more than two data source nodes, and the metadata is used to describe the storage information of the original data in the data nodes, and then entities and the relationships between entities can be extracted in the metadata to construct a knowledge graph corresponding to the metadata, and then according to the user's data retrieval needs, a corresponding dynamic data knowledge directory is generated in the knowledge graph, and the dynamic data knowledge directory includes target metadata determined by the data retrieval needs in the metadata; thus, in the above scheme, metadata is collected and stored for the original data distributedly stored in the data source nodes, thereby eliminating the need to store data of a large number of edge nodes. Data is centrally transmitted to the data center, which reduces the cost of data replication, transmission, and storage, improves data governance efficiency, reduces network congestion, bandwidth costs, and the computing power requirements of the data center, and avoids security and privacy risks that may be caused during data replication, transmission, and storage, thereby improving overall data processing, access, and management. By collecting metadata on the original data and building a knowledge graph based on metadata, cross-business and cross-system data integration between data source nodes can be achieved, thereby improving the global correlation of data. In addition, a dynamic data knowledge catalog can be generated for users' real-time data retrieval needs, so as to provide a unified, comprehensive, and intuitive data view, and achieve accurate retrieval and access to data. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 One of the step flow charts of the data weaving processing method provided in the embodiment of the present disclosure;

[0027] Figure 2 The second step flow chart of the data weaving processing method provided by the embodiment of the present disclosure;

[0028] Figure 3 A flowchart of the implementation framework of the data weaving processing method provided in the embodiment of the present disclosure;

[0029] Figure 4 A data weaving processing architecture provided for an embodiment of the present disclosure;

[0030] Figure 5 This is one of the structural schematic diagrams of the data weaving processing device provided in the embodiment of the present disclosure;

[0031] Figure 6 The second structural diagram of the data weaving processing device provided by the embodiment of the present disclosure;

[0032] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure;

[0033] Figure 8 A hardware schematic diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0035] The terms "first", "second", etc. in the specification and claims of the present disclosure are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable when appropriate, so that the embodiments of the present disclosure can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0036] The data weaving processing method provided by the embodiment of the present disclosure is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0037] Figure 1 This is one of the step flow charts of the data weaving processing method provided by the embodiment of the present disclosure. Figure 1 As shown, the method may include the following steps 101 to 104.

[0038] Step 101: collect and store metadata corresponding to the original data. The original data is distributedly stored in more than two data source nodes. The metadata is used to describe the storage information of the original data in the distributed storage.

[0039] Among them, the original data can be related data that is distributedly stored between data source nodes, and can include user data, business data, network data, etc. For example, the original data can include B (Business Support System, business support system) domain data, M (Management Support System, management support system) domain data, O (Operation Support System, operation support system) domain data, DPI (Deep Packet Inspection, deep data packet inspection) data, enterprise data, industry data, etc. The original data is generated and stored in the corresponding local data source node. The data source node can include a local data warehouse or a cloud data warehouse, etc.

[0040] In the disclosed embodiment, metadata corresponding to the original data can be collected and stored from the data source node. Metadata can describe the original data, such as obtaining the storage information of the original data in the data source node, and by describing the storage location and context corresponding to the original data, the organization, search and understanding of the original data can be achieved. In the data weaving processing method provided in the disclosed embodiment, metadata can be collected for the original data distributedly stored between local data source nodes through ETL (Extract-Transform-Load), and after the metadata is obtained by extracting, cleaning, transforming and loading the heterogeneous multi-source original data, the collected metadata can be stored.

[0041] In a method embodiment of the present disclosure, based on the heterogeneous characteristics of metadata, structured metadata, semi-structured metadata, and unstructured metadata can be stored in a source data management warehouse of corresponding characteristics, such as field name metadata is structured metadata, which can be stored in a relational database; form metadata is semi-structured metadata, which can be stored in a non-relational database; log metadata is unstructured metadata, which can be stored in a file storage warehouse. By obtaining corresponding metadata for distributed stored raw data, and uniformly aggregating and storing the metadata, it is convenient to centrally manage and weave the scattered and edge raw data, and there is no need to copy and transmit the full amount of raw data to the central data lake, which reduces the risk of data privacy leakage, shortens data turnover time, and can also avoid possible write errors when the raw data enters the lake, reducing operation and maintenance costs.

[0042] It should be noted that the above-mentioned metadata and original data in this disclosure are accessed, collected, stored and used for subsequent analysis and processing with the user's consent and authorization, after the user is clearly informed of the collection content, data purpose, processing method and other information, and the user can be provided with ways to access, correct and delete the data, as well as methods to revoke consent and authorization.

[0043] Step 102: Extract entities and relationships between entities from the metadata, and construct a knowledge graph corresponding to the metadata based on the entities and relationships.

[0044] Among them, the knowledge graph is used to achieve virtual full linking of original data that are geographically and business-isolated between different data source nodes. Specifically, entities and relationships between entities can be extracted from metadata corresponding to the original data. Entities can be abstract objects such as people, things, and objects in the data, usually nouns, and relationships can refer to the way and degree of association between entities. Nodes in the knowledge graph are constructed based on entities, and edges between nodes in the knowledge graph are constructed based on relationships between entities, thereby forming a knowledge graph that links heterogeneous multi-source metadata based on association relationships.

[0045] In a method embodiment of the present disclosure, for various types of complex metadata, entities, concepts, and relationships between entities can be extracted based on the business type and association rules to which they belong, and then a relational data model that can represent data relationships can be constructed. Based on the relational data model, entities and relationships between entities can be calculated to construct nodes and edges connecting nodes, forming a fully linked mesh knowledge structure to obtain a knowledge graph corresponding to the metadata.

[0046] Step 103: Determine the user's data retrieval requirements.

[0047] Among them, the user can be a data producer, data consumer, platform manager, etc., and can receive data retrieval information such as voice, characters, images, etc. input by the user, and parse the data retrieval information to determine the user's data retrieval needs, which can be determined in the form of receiving keywords provided by the user or questions raised. The user's data retrieval needs can be to obtain original data, or derived needs for original data, such as derived needs for original data can include determining the data source node corresponding to the original data, the association relationship between the original data, and the analysis results of the original data based on the demand business.

[0048] Step 104: Generate a dynamic data knowledge catalog in the knowledge graph according to data retrieval requirements, and the dynamic data knowledge catalog includes target metadata determined in the metadata according to data retrieval requirements.

[0049] Among them, based on the user's data retrieval needs, the target metadata that meets the data retrieval needs can be determined in the knowledge graph. The knowledge graph represents the entities in the full amount of metadata and the relationships between the entities, so that the target metadata corresponding to the data retrieval needs can be accurately and quickly located through the knowledge graph. By generating a dynamic data knowledge catalog including the target metadata, when the original data is not local, the user does not need to understand the basic meaning of each metadata, and can realize cross-domain, cross-business, and cross-system knowledge links through the knowledge graph, and mine the full-domain business data that cannot be covered by humans, and provide users with accurate retrieval results including relationships, weights and other information in real time and dynamically.

[0050] In the disclosed embodiment, the dynamic data knowledge directory refers to a knowledge directory that is dynamically generated according to the user's data retrieval needs and includes knowledge representations corresponding to target metadata. According to changes in data retrieval needs, the target metadata corresponding to the dynamic data knowledge directory also changes, so that the dynamic data knowledge directory changes accordingly, such as when the target metadata decreases, the dynamic data knowledge directory deletes nodes or edges, when the target metadata increases, the dynamic data knowledge directory adds nodes or edges, and when the target metadata is relocated, the dynamic data knowledge directory is reconstructed. The dynamic data knowledge directory is adapted to the user's current data retrieval needs, consistent with the user scope of the user, and matches the business needs. It can be generated in real time according to the user's data retrieval needs, and updated or reconstructed.

[0051] In the data weaving processing method provided by the present disclosure, metadata corresponding to the original data can be collected, wherein the original data is distributedly stored in more than two data source nodes, and the metadata is used to describe the storage information of the original data in the data nodes, and then entities and the relationships between entities can be extracted in the metadata to construct a knowledge graph corresponding to the metadata, and then according to the user's data retrieval needs, a corresponding dynamic data knowledge directory is generated in the knowledge graph, and the dynamic data knowledge directory includes target metadata determined by the data retrieval needs in the metadata; thus, in the above scheme, metadata is collected and stored for the original data distributedly stored in the data source nodes, thereby eliminating the need to store data of a large number of edge nodes. Data is centrally transmitted to the data center, which reduces the cost of data replication, transmission, and storage, improves data governance efficiency, reduces network congestion, bandwidth costs, and the computing power requirements of the data center, and avoids security and privacy risks that may be caused during data replication, transmission, and storage, thereby improving overall data processing, access, and management. By collecting metadata on the original data and building a knowledge graph based on metadata, cross-business and cross-system data integration between data source nodes can be achieved, thereby improving the global correlation of data. In addition, a dynamic data knowledge catalog can be generated for users' real-time data retrieval needs, so as to provide a unified, comprehensive, and intuitive data view, and achieve accurate retrieval and access to data.

[0052] Figure 2 This is a second flow chart of the steps of the data weaving processing method provided by the embodiment of the present disclosure. Figure 2 As shown, the method may include the following steps 201 to 216.

[0053] Step 201: collect and store metadata corresponding to the original data. The original data is distributedly stored in more than two data source nodes. The metadata is used to describe the storage information of the original data in the distributed storage.

[0054] In the embodiment of the present disclosure, step 201 may correspond to the relevant description of the aforementioned step 101, and will not be described again here to avoid repetition.

[0055] Figure 3 The following is a flowchart of the implementation framework of the data weaving processing method provided in the embodiment of the present disclosure. Figure 3 As shown, the full amount of original data includes the B, M, and O domain data of the telecom operator, which is distributedly stored in DC (Data Center) 1, DC2, DC3, DC4, and DC5. In the disclosed embodiment, in step ①, the following metadata is collected from DC1, DC2, DC3, DC4, and DC5, and the collected metadata is stored in the metadata management warehouse.

[0056]

[0057] Step 202: Classify and manage the metadata, where the classification management includes one or more of metadata statistics, metadata model definition, metadata classification rule definition, metadata version management, and metadata change management.

[0058] Among them, in the metadata management warehouse storing metadata, classification management can be performed according to the management requirements of metadata. Specifically, metadata can be classified according to the requirements first, and then metadata can be further managed according to the obtained classification. For example, the classification of metadata can be based on metadata statistics, metadata model definition, metadata classification rule definition, etc. Metadata statistics can include statistics on metadata quantity, source, path and other information; metadata model can define the grammatical structure and semantics of metadata, and unified management can be achieved by using the defined metadata model description for heterogeneous and multi-source metadata; metadata classification rule definition can define different metadata classification methods according to requirements, metadata classification rules can integrate metadata conversion operations, preset conditions, etc., metadata can be divided into business metadata, technical metadata, operational metadata, etc. according to data types, among which operational metadata and business metadata parts can be presented in text forms such as job execution logs, research reports, business rules, etc. Metadata management based on classification can include metadata version management and metadata change management. Metadata version management can include the addition, modification, deletion, release, status change, call permission of metadata versions, as well as query, comparison and rollback of metadata versions; metadata change management can include the review and tracing of metadata changes.

[0059] In a method embodiment of the present disclosure, the metadata of the classification management can also be displayed, and data query based on the metadata can be realized. On this basis, the impact analysis, correlation analysis, lineage analysis and other processing can also be performed based on the metadata, such as analysis of the content, ownership, source, traceability, change history, etc. of the data, and further corresponding analysis results can be provided in the data query.

[0060] like Figure 3 As shown, in step ②, a metadata model (Metadata-Model) is constructed in the metadata management warehouse, and metadata is described based on the metadata model, and then classified and managed.

[0061] Step 203: Divide the metadata into first metadata and second metadata.

[0062] Among them, in the process of constructing the knowledge graph, a preliminary construction can be first performed based on a part of the metadata to obtain a knowledge graph with high accuracy but not easy to expand, and then based on another part of the metadata, the knowledge graph is supplemented, expanded and incrementally updated to obtain the knowledge graph corresponding to the full amount of metadata. Therefore, the metadata can be divided into first metadata and second metadata. The division of the first metadata and the second metadata can be set according to the actual data volume, application requirements, data quality, etc., and can be divided according to the type and field of metadata, or can be randomly divided, that is, the first metadata and the second metadata can be different metadata in the same field, metadata in different fields, different metadata of the same type, metadata of different types, etc., and the embodiments of the present disclosure do not impose specific restrictions on this.

[0063] In a method embodiment of the present disclosure, metadata may be cleaned, preprocessed, etc. to improve the accuracy and reliability of constructing a knowledge graph based on the metadata.

[0064] Step 204: Identify first entities in the first metadata, and extract first relationships between the first entities in the first metadata.

[0065] The first entities in the first metadata and the relationships between the first entities may be identified and extracted, which may be achieved through manual annotation or by using entity recognition algorithms, relationship mining algorithms, and the like.

[0066] Step 205: construct a first knowledge representation corresponding to the first metadata according to the first entity and the first relationship.

[0067] Among them, the first knowledge representation corresponding to the first metadata can be further constructed according to the first entity and the first relationship. On the basis that the first metadata is partial metadata, the first knowledge representation has a high accuracy in representing entities and relationships between entities, but is not easy to expand.

[0068] Step 206: Update the first knowledge representation according to the second metadata to obtain a knowledge graph corresponding to the metadata.

[0069] In the disclosed embodiment, based on the first knowledge representation, the second metadata can be used for updating, so as to complete and expand entities and relationships, realize incremental updating of knowledge, and obtain the knowledge graph corresponding to the metadata.

[0070] In a method embodiment of the present disclosure, step 206 may include sub-steps S11 to S14, as shown below:

[0071] Step S11: Identify second entities in the second metadata, and extract second relationships between the second entities in the second metadata.

[0072] Among them, the second metadata can first be used to identify and extract the second entity and the second relationship between the second entities. According to its structural characteristics, the second metadata can include structured metadata, semi-structured metadata and unstructured metadata. For structured metadata, the relationship between the second entity and the second entity can be directly determined based on its structural characteristics; for semi-structured metadata and unstructured metadata, the extraction of entities, relationships and attributes can be achieved based on entity recognition, relationship mining rules, machine learning algorithms, deep learning algorithms and other methods.

[0073] Step S12: performing entity alignment on the second entity, where the entity alignment includes one or more of entity disambiguation and coreference resolution.

[0074] Among them, based on obtaining the second entity from the second metadata, knowledge fusion can be further achieved. Specifically, entity alignment can be performed through entity disambiguation, coreference resolution, etc. to eliminate redundant information and erroneous data contained therein.

[0075] Step S13: Add the second entity and the second relationship to the first knowledge representation to obtain a second knowledge representation.

[0076] Among them, after realizing entity alignment for the second entity, the second entity and the second relationship can be added to the first knowledge representation. For example, the synonymous first entity and second entity can be merged, or a corresponding node can be constructed for the new second entity. Based on the second relationship, new edges can be connected between the first entities, between the second entities, and between the first entity and the second entity to realize the expansion of the first knowledge representation and obtain the second knowledge representation constructed based on the first metadata and the second metadata.

[0077] Step S13: Use the knowledge representation learning model in the second knowledge representation to perform entity prediction and relationship prediction to obtain a knowledge graph.

[0078] Among them, the knowledge representation learning model is used to learn the representations between entities and relationships, so as to efficiently calculate the semantic associations of entities and relationships in the knowledge base, and thus predict unknown entities and relationships based on the semantic associations. In the disclosed embodiment, based on the second knowledge representation, the knowledge representation learning model can be used to perform entity prediction and relationship prediction, and to mine possible relationships and entities in the second knowledge representation to achieve further completion, ensure the globality and accuracy of the knowledge graph, and obtain the knowledge graph corresponding to the full metadata.

[0079] like Figure 3 As shown in the figure, in step ③, entities and relationships between entities are extracted from metadata, and knowledge representation is constructed based on entities and relationships. The knowledge representation is further reasoned, completed and updated to construct a knowledge graph corresponding to the metadata.

[0080] Step 207: Associate the knowledge graph with the original data based on the correspondence between the metadata and the original data.

[0081] Among them, the knowledge graph and the original data can be associated based on the correspondence between the metadata and the original data. Specifically, the local location and access rights can be obtained through metadata in the distributed physical servers, distributed cloud servers and other data source nodes that store the original data, so as to match the link between the metadata and the original data, and establish an index between the metadata and the original data based on the link, and then map the index to the location information of the original data associated with the knowledge graph, so as to realize the association between the knowledge graph and the original data. Based on this association, data services related to the original data can be provided through the knowledge graph.

[0082] Step 208: Build a static data knowledge directory corresponding to the metadata based on the knowledge graph.

[0083] Among them, a static data knowledge directory can be constructed on the knowledge graph. The static data knowledge directory can be constructed based on the full amount of original data. According to the position and connection relationship of nodes and edges in the knowledge graph, and the application requirements of the static data knowledge directory, a hierarchical static data knowledge directory can be constructed according to the hierarchical relationship between different nodes in the knowledge graph, or a static data knowledge directory corresponding to a separate business or a separate industry can be constructed based on business type, industry type, etc.

[0084] In a method embodiment of the present disclosure, the knowledge graph and the original data can be associated based on the corresponding relationship between the metadata and the original data. Specifically, the local location and access rights can be obtained through the metadata in the distributed physical server, distributed cloud server and other data source nodes storing the original data, so as to match the link between the metadata and the original data, and establish an index between the metadata and the original data based on the link, and then map the index to the location information of the original data associated with the knowledge graph, so as to achieve the association between the knowledge graph and the original data. Based on the association, data services related to the original data can be provided through the knowledge graph.

[0085] Step 209: Determine the user's data retrieval requirements.

[0086] In the embodiment of the present disclosure, step 209 may correspond to the relevant description of the aforementioned step 103, and will not be described again here to avoid repetition.

[0087] like Figure 3 As shown, in step ④, the data search information provided by the data user (Data User) through the personal service data search (Self-Service Data Search) can be obtained to determine the user's data search needs.

[0088] Step 210: Generate a dynamic data knowledge catalog in the knowledge graph according to data retrieval requirements, and the dynamic data knowledge catalog includes target metadata determined in the metadata according to the data retrieval requirements.

[0089] In the embodiment of the present disclosure, step 210 may correspond to the relevant description of the aforementioned step 104, and will not be described again here to avoid repetition.

[0090] like Figure 3 As shown, in step ⑤, an enhanced knowledge catalog (Enhanced Knowledge Catalog) including target metadata can be constructed based on the knowledge graph based on data retrieval requirements, wherein the enhanced knowledge catalog can be a dynamic data knowledge catalog that changes based on changes in data retrieval requirements.

[0091] In a method embodiment of the present disclosure, based on determining the target metadata, the association relationship can be further expanded to obtain the associated metadata corresponding to the target metadata, so as to provide a broader and more comprehensive data map, as shown in the following steps 211 to 212.

[0092] Step 211: Acquire associated metadata corresponding to the target metadata according to a preset association relationship, where the preset association relationship includes one or more of a causal association relationship, a correlation association relationship, and an integrated association relationship.

[0093] Among them, the preset association relationship can be the association relationship between other metadata and the target metadata in addition to the target metadata, so as to distinguish it from the relationship between the target metadata, that is, the relationship between the target metadata can be regarded as the first-level association relationship, and the preset association relationship can be regarded as the second-level association relationship. The second-level association relationship and the first-level association relationship can be the same-layer association relationship or the upper-lower-layer association relationship. The preset association relationship can be selected according to actual needs, and can include causal association relationship, correlation association relationship, and integrated association relationship. The causal association relationship refers to the three-level association relationship between the associated metadata and the target metadata, namely, the variables of the associated metadata and the target metadata change synchronously, and the variables of the associated metadata change accordingly when the variables of the target metadata are changed, and the variables of the associated metadata can be changed as expected by changing the variables of the target metadata; the correlation association relationship refers to the mutual dependence between the associated metadata and the target metadata, such as the associated metadata increases with the increase of the variable of the target metadata, or decreases with the decrease, which is a positive correlation, or the associated metadata decreases with the increase of the variable of the target metadata, or decreases with the increase of the variable of the target metadata, which is a negative correlation; the integrated relationship can refer to the integration of metadata and associated metadata based on business, application, management and other needs.

[0094] Step 212: Provide a data map to the user based on the target metadata and the associated metadata.

[0095] Among them, a data map can be constructed according to the target metadata and the associated metadata to represent the hierarchy, path, etc. of the relationship links between the target metadata and between the target metadata and the associated metadata.

[0096] Taking the application scenario of telecom operator business billing analysis as an example, through the data retrieval information entered by the user, it can be determined that the user's data retrieval demand is "this month's business billing analysis". At this time, there is no need to perform data query methods such as table join operations. The target metadata corresponding to "this month's business billing analysis" can be determined in the data location range corresponding to the user in the knowledge graph, such as "free machine building income", "cooperative machine building income", "rack income", "bandwidth income", "other income" and other data fields to generate the corresponding dynamic data knowledge directory. Furthermore, according to the preset association relationship, the associated metadata corresponding to the target metadata can be obtained, such as "own machine building broadband inflow traffic", "cooperative machine building broadband traffic", "rack broadband traffic", "bandwidth type broadband total traffic", "other broadband total traffic" and other data fields, so as to construct the corresponding data map.

[0097] In a method embodiment of the present disclosure, on the basis of determining the target metadata, the target data described by the target metadata can be further determined in the original data, and then the target data can be analyzed and processed to meet specific data service requirements. As shown in steps 213 to 216 below:

[0098] Step 213: Acquire corresponding target data and a data source node corresponding to the target data in the original data according to the target metadata.

[0099] Among them, on the basis of determining the target metadata based on the data retrieval requirement, the target metadata can be mapped to the corresponding data source node, so as to determine the target data that meets the data retrieval requirement.

[0100] like Figure 3 As shown, in step ⑥, after determining the target metadata based on the data retrieval requirement "this month's business bill analysis", the target metadata can be traced back to the corresponding data source nodes DC1 and DC5. The metadata includes storage information describing the original data attributes such as database objects, field locations, access permissions, and processing logs. Therefore, the target metadata can be traced back to the corresponding two distributed data source nodes DC1 and DC5.

[0101] Step 214: Obtain a corresponding target algorithm according to data retrieval requirements and target data.

[0102] Among them, after determining the target data, the data retrieval requirements and the target algorithm corresponding to the target data can be scheduled to realize the analysis and processing of the target data locally at the data source node. In the disclosed embodiment, a variety of heterogeneous algorithm models can be pre-configured, and different algorithm models can be disassembled and aggregated to realize different data processing tasks, such as recognition, conversion, classification, prediction, etc. of voice data, text data, image data, behavior data, etc., so as to meet the analysis requirements of behavior paths, demand mining, value assessment and other businesses. The target data is parsed at the corresponding data source node based on the target algorithm, so that the analysis results that are compatible with the data retrieval requirements are obtained without the target data leaving the local area, thereby ensuring the security and reliability of the data source and realizing the actual needs of data retrieval.

[0103] Step 215: Decompose the target algorithm into the data source nodes corresponding to the target data, and perform joint training between the data source nodes based on federated learning to parse the target data.

[0104] Among them, since the original data is heterogeneous and multi-source, it is distributed and stored in more than two data source nodes. Therefore, the target data may be distributed in more than two data source nodes, so joint training can be performed between different data source nodes based on federated learning. For example, the model parameters of the target algorithm can be distributed to each data source node to decompose the target algorithm into the corresponding data source node, and gradient convergence can be performed to calculate the loss function, and joint training can be performed between data source nodes based on federated learning, and the distribution and convergence of model parameters are repeated until the model converges.

[0105] In one embodiment of the present disclosure, when data sharing is required among multiple data source nodes, data scheduling can be performed based on the data ownership confirmation protocol, output data storage location, computing type and other information of each data source node, thereby achieving secure data sharing, wherein the computing type may include general privacy computing, TEE (Trusted Execution Environment), etc.

[0106] Further, such as Figure 3 As shown, in step ⑦, the algorithm corresponding to the target data that meets the data retrieval requirements is matched, and the algorithm is jointly trained between DC1 and DC5 based on federated learning to obtain the parsing result.

[0107] Step 216: Provide analysis results of the target data based on the target algorithm.

[0108] Among them, after the original data is locally jointly trained at the data source node through the target algorithm to obtain the analysis result, the analysis result of the target data based on the target algorithm can be provided to the user. For example, the analysis result can be directly provided, such as data classification, prediction results, etc., and further application analysis, execution decision, etc. can be carried out based on the analysis result.

[0109] like Figure 3 As shown, taking user insight (Insight) in the telecommunications field as an example, based on the analysis results of step ⑦, in step ⑧, the behavior characteristics, consumption habits and other data of telecommunications users can be sorted and described, or the telecommunications users can be further grouped based on the behavior characteristics, consumption habits, etc., so as to provide service plan planning for telecommunications user groups and achieve differentiated decision-making recommendations.

[0110] In the disclosed embodiment, the global association of business needs is realized, the target data is accurately located, and the original data can be managed intelligently and comprehensively. The data consumption process includes "getting data, managing data, knowing data, and using data". Through metadata collection, metadata management, metadata knowledge graph construction, data location analysis based on knowledge graph, etc., intelligent self-service data consumption is provided to users.

[0111] like Figure 3 As shown, after determining the user's data retrieval needs, the analysis results of the data retrieval needs or further planning suggestions based on the analysis results can be obtained through the aforementioned steps ② to ⑦, and then the analysis results or planning suggestions can be provided to the user in step ⑨ to provide the user with self-service, imperceptible data consumption (Data Consumption).

[0112] In the data weaving processing method provided by the present disclosure, metadata corresponding to the original data can be collected, wherein the original data is distributedly stored in more than two data source nodes, and the metadata is used to describe the storage information of the original data in the data nodes, and then entities and the relationships between entities can be extracted in the metadata to construct a knowledge graph corresponding to the metadata, and then according to the user's data retrieval needs, a corresponding dynamic data knowledge directory is generated in the knowledge graph, and the dynamic data knowledge directory includes the target metadata determined by the data retrieval needs in the metadata. Furthermore, the original data can be associated on the basis of the knowledge graph to construct a static data knowledge directory corresponding to the original data; a target algorithm corresponding to the data retrieval needs can also be provided, and the target algorithm can be used to parse the target data in the corresponding data source node based on the target algorithm, and the parsing results can be provided to the user; the collected metadata can also be classified and managed.

[0113] In the above scheme, metadata is collected and stored for the original data distributedly stored in the data source nodes, so that there is no need to centrally transmit the data of a large number of edge nodes to the data center, reducing the cost of data replication, transmission and storage, improving data governance efficiency, reducing network congestion, bandwidth costs, and computing power requirements of the data center, and avoiding security and privacy risks that may be caused during data replication, transmission and storage, and improving the overall data processing, access and management work; by realizing metadata collection on the original data and building a knowledge graph based on metadata, cross-business and cross-system data integration between data source nodes can be achieved, and the global correlation of data can be improved; by building a knowledge graph corresponding to metadata and generating a static data knowledge directory of the entire original data based on the knowledge graph, and a dynamic data knowledge directory corresponding to data retrieval requirements, a unified, comprehensive and intuitive business data view can be provided to users; by matching the target algorithm that is compatible with the data retrieval requirements, local data analysis is performed at the data source node corresponding to the target data, and cross-business and cross-system data analysis can be provided without leaving the local area, while ensuring the credibility and security of the data source, the full use of data is achieved, and it is convenient to provide users with information links that cannot be achieved manually, mine the hidden information of isolated island data, grasp the hidden attribute relationship between businesses, and achieve data insight.

[0114] Figure 4A data weaving processing architecture 400 is provided in an embodiment of the present disclosure. The architecture is divided into functional layers based on each step in the above-mentioned data weaving processing method, as shown below.

[0115] like Figure 4 As shown, the architecture may include a collection and storage layer 401, a data management layer 402, a knowledge graph layer 403, a data directory layer 404, a model application layer 405, and a user demand layer 406.

[0116] The collection storage layer 401 can be used to collect and store metadata corresponding to the original data. The original data is distributedly stored in more than two data source nodes, and the metadata is used to describe the storage information of the original data in the data source nodes. The collection storage layer 401 can be divided into two functional units, metadata collection 4011 and metadata storage 4012, based on the functions implemented. The metadata collection 4011 can collect metadata corresponding to the original data, which can be implemented based on the corresponding metadata collection tool; the metadata storage 4012 can store the collected metadata, which can be implemented based on the corresponding database.

[0117] The data management layer 402 can be used to classify and manage the metadata, and the classification management includes one or more of metadata statistics, metadata model definition, metadata classification rule definition, metadata version management, and metadata change management. The data management layer 402 can be divided into two functional units, metadata model 4021 and metadata display 4022, based on the functions it implements. The metadata model 4021 can implement the construction, application, and update of the metadata model to obtain metadata that meets business requirements and specifications. The metadata display 4022 can display metadata, or further display the results of metadata query, impact analysis, correlation analysis, lineage analysis, etc.

[0118] The knowledge graph layer 403 can be used to extract entities and relationships between entities in metadata, and construct a knowledge graph corresponding to the metadata based on the entities and relationships. Among them, the knowledge graph layer 403 can be divided into seven functional units based on the functions it implements: entity recognition 4031, relationship extraction 4032, entity disambiguation 4033, reference resolution 4034, weight analysis 4035, knowledge reasoning 4036, and rule reasoning 4037. Entity recognition 4031 can identify entities in metadata based on semantic features, relationship extraction 4032 can extract relationships for identified entities, entity disambiguation 4033 can identify different entities with the same reference, reference resolution 4034 can identify different references for the same entity, weight analysis 4045 can measure the strength of the relationship between entities, knowledge reasoning 4046 mines and infers new knowledge to complete the knowledge graph, and rule reasoning 4047 mines and infers new rules to complete the knowledge graph. The above functional units can be implemented by configuring algorithms for corresponding functions.

[0119] The data directory layer 404 can be used to implement the index association of metadata, trace the data path, and generate an enhanced knowledge directory linked to the original data based on the knowledge graph. Among them, the data directory layer 404 can be divided into four functional units based on the functions it implements, namely, data index association 4041, data assets 4042, tools 4043, and activities 4044. The data assets 4042 can be divided into two functional sub-units, namely, static data knowledge directory 40421 and dynamic data knowledge directory 40422. Data index association 4041 can associate metadata and its corresponding original data; data assets 4042 are used to build an enhanced knowledge directory based on the knowledge graph, including building a hierarchical or static data knowledge directory corresponding to different classifications of original data in the static data knowledge directory 40421, and dynamically and real-time building a dynamic data knowledge directory corresponding to data retrieval needs in the dynamic data knowledge directory 40422; tools 4043 can realize data insight, data path, and data value analysis based on the knowledge graph; activities 4044 are used to record and display user operation data, log data, etc.

[0120] The model application layer 405 can be used to match the corresponding target algorithm according to the data retrieval requirements, and parse the target data at the corresponding data source node based on the target algorithm to provide the user with the parsing results of the target algorithm. Among them, the model application layer 405 can be divided into four functional units based on the functions it implements: algorithm matching 4051, algorithm decomposition 4052, federated learning 4053, and analysis results 4054. Algorithm matching 4051 can match the target algorithm to the target data according to the data retrieval requirements, algorithm decomposition 4042 can distribute and aggregate model parameters to decompose the target algorithm to the data source node corresponding to the target data, federated learning 4043 can realize joint training between distributed data source nodes, and analysis results 4044 can obtain and display the analysis results of the target algorithm on the target data, or can further evaluate the analysis results.

[0121] The user requirement layer 406 can be used to obtain the user's data retrieval requirements and provide the user with analysis results corresponding to the data retrieval requirements. The user requirement layer 406 can be divided into five functional units based on the functions it implements: user authority 4061, user search 4062, data map 4063, algorithm map 4064, and decision view 4065. Among them, user permissions 4061 can be used to allocate, restrict and manage permissions for users, and permissions can be data ownership, data usage rights, data management rights, algorithm ownership, algorithm usage rights, algorithm management rights, etc.; user search 4062 can receive user input and determine its data retrieval requirements based on the user input, and can also provide its determined data retrieval requirements to other functional layers, such as model application layer 405, data directory layer 404, etc.; data map 4063 can provide a data map corresponding to the target metadata and the associated metadata to achieve holistic and comprehensive data insights; algorithm map 4064 can provide algorithm maps to users based on algorithm attributes to display algorithm attributes such as algorithm corresponding workflow, analysis purpose, algorithm category, etc., and provide users with an intuitive, open, shared, and easy-to-iterate algorithm map application mechanism; decision view 4065 can make decisions based on the analysis results of the target algorithm, data retrieval requirements, etc., to assist users in business practices.

[0122] It should be noted that the division of the above-mentioned functional layers, functional units, and functional sub-units is only used to distinguish different execution logics and implement functions, but does not limit the specific implementation method. For example, different servers, networks, algorithm models, etc. and their combinations can be used to implement the above-mentioned data weaving processing architecture, thereby executing the data weaving processing method provided by the present disclosure.

[0123] The data weaving processing method provided in the embodiment of the present disclosure can also be executed by a data weaving processing device, or a control module in the data weaving processing device for executing the data weaving processing method. In the embodiment of the present disclosure, the method of executing data weaving processing by a data weaving processing device is taken as an example to illustrate the data weaving processing device provided in the embodiment of the present disclosure.

[0124] Figure 5 This is one of the structural schematic diagrams of the data knitting processing device 500 provided in the embodiment of the present disclosure, such as Figure 5As shown in , the data weaving processing device may include an acquisition and storage module 501, a knowledge graph module 502, a user demand module 503 and a data directory module 504; wherein, the acquisition and storage module 501 is used to acquire and store metadata corresponding to the original data, the original data is distributedly stored in more than two data source nodes, and the metadata is used to describe the storage information of the original data in the distributed storage; the knowledge graph module 502 is used to extract entities and the relationships between entities in the metadata, and construct a knowledge graph corresponding to the metadata based on the entities and relationships; the user demand module 503 is used to determine the user's data retrieval needs; the data directory module 504 is used to generate a dynamic data knowledge directory in the knowledge graph according to the data retrieval needs, and the first knowledge directory includes the target metadata determined in the metadata according to the data retrieval needs.

[0125] In an apparatus embodiment of the present disclosure, the knowledge graph module 502 includes a data partitioning submodule, a first metadata processing submodule, a first knowledge representation construction submodule and a knowledge graph updating submodule; wherein the data partitioning submodule is used to partition the metadata into first metadata and second metadata; the first metadata processing submodule is used to identify the first entity in the first metadata and extract the first relationship between the first entities in the first metadata; the first knowledge representation construction submodule is used to construct the first knowledge representation corresponding to the first metadata according to the first entity and the first relationship; the knowledge graph updating submodule is used to update the first knowledge representation according to the second metadata to obtain the knowledge graph corresponding to the metadata.

[0126] In an apparatus embodiment of the present disclosure, a knowledge graph update submodule includes a second metadata processing unit, an entity alignment unit, a second knowledge representation construction unit, and a knowledge graph prediction update unit; wherein the second metadata processing unit is used to identify second entities in the second metadata and extract second relationships between second entities in the second metadata; the entity alignment unit is used to perform entity alignment on the second entities, and entity alignment includes one or more of entity disambiguation and coreference resolution; the second knowledge representation construction unit is used to add the second entity and the second relationship to the first knowledge representation to obtain a second knowledge representation; the knowledge graph prediction update unit is used to perform entity prediction and relationship prediction in the second knowledge representation using a knowledge representation learning model to obtain a knowledge graph.

[0127] In an apparatus embodiment of the present disclosure, the data directory module 504 is also used to associate the knowledge graph with the original data based on the correspondence between the metadata and the original data, and to construct a static data knowledge directory corresponding to the metadata based on the knowledge graph.

[0128] In an apparatus embodiment of the present disclosure, the user demand module 503 is also used to obtain associated metadata corresponding to the target metadata based on a preset association relationship, and provide a data map to the user based on the target metadata and the associated metadata. The preset association relationship includes one or more of a causal association relationship, a correlation association relationship, and an integrated association relationship.

[0129] In the data weaving processing device provided by the present disclosure, metadata corresponding to the original data can be collected through the collection and storage module, wherein the original data is distributedly stored in more than two data source nodes, and the metadata is used to describe the storage information of the original data in the data nodes, and then the entities and the relationships between the entities can be extracted in the metadata through the knowledge graph module to construct a knowledge graph corresponding to the metadata, and then according to the user's data retrieval requirements provided by the user requirement module, a corresponding dynamic data knowledge directory is generated in the knowledge graph through the data directory module, and the dynamic data knowledge directory includes the target metadata determined by the data retrieval requirements in the metadata; thus, in the above scheme, metadata is collected and stored in the original data source nodes in a distributed manner. Storage eliminates the need to centrally transmit data from a large number of edge nodes to the data center, reducing the cost of data replication, transmission, and storage, improving data governance efficiency, reducing network congestion, bandwidth costs, and the computing power requirements of the data center, and avoiding security and privacy risks that may be caused during data replication, transmission, and storage, thereby improving overall data processing, access, and management. By collecting metadata on the original data and building a knowledge graph based on metadata, cross-business and cross-system data integration between data source nodes can be achieved, thereby improving the global correlation of data. In addition, a dynamic data knowledge catalog can be generated for users' real-time data retrieval needs to provide a unified, comprehensive, and intuitive data view, thereby achieving accurate retrieval and access to data.

[0130] Figure 6 The second structural diagram of the data knitting processing device 600 provided in the embodiment of the present disclosure is as follows: Figure 6As shown in , the data weaving processing device may include an acquisition and storage module 601, a data management module 602, a knowledge graph module 603, a user demand module 604, a data directory module 605 and a model application module 606; wherein the acquisition and storage module 601 is used to acquire and store metadata corresponding to the original data, the original data is distributedly stored in more than two data source nodes, and the metadata is used to describe the storage information of the original data in the distributed storage; the data management module 602 is used to classify and manage the metadata, and the classification management includes one or more of metadata statistics, metadata model definition, metadata classification rule definition, metadata version management, and metadata change management; the knowledge graph module 603 is used to extract entities in the metadata, as well as the relationship between entities, And construct a knowledge graph corresponding to the metadata based on the entities and relationships; a user demand module 604 is used to determine the user's data retrieval needs; a data directory module 605 is used to generate a dynamic data knowledge directory in the knowledge graph according to the data retrieval needs, and the first knowledge directory includes the target metadata determined in the metadata according to the data retrieval needs; a model application module 606 is used to obtain the corresponding target data in the original data according to the target metadata, and the data source node corresponding to the target data, and obtain the corresponding target algorithm according to the data retrieval needs and the target data, and then decompose the target algorithm into the data source node corresponding to the target data, and perform joint training between the data source nodes based on federated learning to parse the target data, and provide the parsing results of the target data based on the target algorithm.

[0131] In the data weaving processing device provided by the present disclosure, metadata corresponding to the original data can be collected through the collection and storage module, wherein the original data is distributedly stored in more than two data source nodes, and the metadata is used to describe the storage information of the original data in the data node, and then the entities and the relationship between the entities can be extracted in the metadata through the knowledge graph module to construct the knowledge graph corresponding to the metadata, and then according to the user's data retrieval requirements provided by the user requirement module, the corresponding dynamic data knowledge directory is generated in the knowledge graph based on the data directory module, and the dynamic data knowledge directory includes the target metadata determined by the data retrieval requirements in the metadata. Further, the data directory module can be used to associate the original data on the basis of the knowledge graph to construct a static data knowledge directory corresponding to the original data; the model application module can also provide a target algorithm corresponding to the data retrieval requirements, and the target algorithm can be used to parse the target data in the corresponding data source node based on the target algorithm, and provide the user with the parsing results; the data management module can also be used to implement classification management of metadata.

[0132] In the above scheme, metadata is collected and stored for the original data distributedly stored in the data source nodes, so that there is no need to centrally transmit the data of a large number of edge nodes to the data center, reducing the cost of data replication, transmission and storage, improving data governance efficiency, reducing network congestion, bandwidth costs, and computing power requirements of the data center, and avoiding security and privacy risks that may be caused during data replication, transmission and storage, and improving the overall data processing, access and management work; by realizing metadata collection on the original data and building a knowledge graph based on metadata, cross-business and cross-system data integration between data source nodes can be achieved, and the global correlation of data can be improved; by building a knowledge graph corresponding to metadata and generating a static data knowledge directory of the entire original data based on the knowledge graph, and a dynamic data knowledge directory corresponding to data retrieval requirements, a unified, comprehensive and intuitive business data view can be provided to users; by matching the target algorithm that is compatible with the data retrieval requirements, local data analysis is performed at the data source node corresponding to the target data, and cross-business and cross-system data analysis can be provided without leaving the local area, while ensuring the credibility and security of the data source, the full use of data is achieved, and it is convenient to provide users with information links that cannot be achieved manually, mine the hidden information of isolated island data, grasp the hidden attribute relationship between businesses, and achieve data insight.

[0133] The data knitting processing device provided by the embodiment of the present disclosure can achieve Figures 1 to 3 To avoid repetition, the various processes implemented in the method embodiment will not be described again here.

[0134] Figure 7 A schematic diagram of the structure of an electronic device 700 provided in an embodiment of the present disclosure is shown in FIG. Figure 7 As shown, the electronic device 700 may include a processor 701, a memory 702, and a program or instruction stored in the memory 702 and executable on the processor 701. When the program or instruction is executed by the processor 701, each process of the above-mentioned data weaving processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0135] It should be noted that Figure 7 The electronic device 700 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0136] Figure 8 A hardware schematic diagram of an electronic device 800 provided in an embodiment of the present disclosure is shown in FIG. Figure 8As shown, the electronic device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to the program stored in ROM (Read Only Memory) 802 or the program loaded from the storage part 808 to RAM (Random Access Memory) 803. In RAM 803, various programs and data required for system operation are also stored. CPU 801, ROM 802 and RAM 803 are connected to each other through bus 804. I / O (Input / Output) interface 805 is also connected to bus 804.

[0137] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

[0138] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 809, and / or installed from a removable medium 811. When the computer program is executed by a central processing unit (CPU801), various functions defined in the system of the present application are executed.

[0139] The embodiment of the present disclosure also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned data weaving processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0140] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as ROM, RAM, magnetic disk or optical disk.

[0141] The disclosed embodiment further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned data weaving processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0142] It should be understood that the chip mentioned in the embodiments of the present disclosure may also be referred to as a system-level chip, a system chip, a chip system, or a system-on-chip chip, etc.

[0143] The embodiment of the present disclosure provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the data weaving processing method as described above and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0144] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiments of the present disclosure is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, an electronic device, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0146] The embodiments of the present disclosure are described above in conjunction with the accompanying drawings, but the present disclosure is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present disclosure, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present disclosure and the claims, all of which are within the protection of the present disclosure.

Claims

1. A data weaving processing method, characterized in that: The method comprises: Collecting and storing metadata corresponding to the original data, the original data is distributedly stored in two or more data source nodes, and the metadata is used to describe storage information of the original data in the distributed storage; Extracting entities and relationships between the entities from the metadata, and constructing a knowledge graph corresponding to the metadata based on the entities and the relationships; Determine the user's data retrieval needs; Generating a dynamic data knowledge catalog in the knowledge graph according to the data retrieval requirement, the dynamic data knowledge catalog including target metadata determined in the metadata according to the data retrieval requirement, the dynamic data knowledge catalog refers to a knowledge catalog dynamically generated according to the user's data retrieval requirement and including knowledge representation corresponding to the target metadata; After generating a dynamic data knowledge catalog in the knowledge graph according to the data retrieval requirement, the method further includes: Acquire corresponding target data and a data source node corresponding to the target data in the original data according to the target metadata; Acquire a corresponding target algorithm according to the data retrieval requirement and the target data; Decomposing the target algorithm into the data source nodes corresponding to the target data, and performing joint training between the data source nodes based on federated learning to parse the target data; Providing analysis results of the target data based on the target algorithm.

2. The method according to claim 1, characterized in that The extracting entities and the relationships between the entities from the metadata, and constructing a knowledge graph corresponding to the metadata according to the entities and the relationships, includes: dividing the metadata into first metadata and second metadata; identifying first entities in the first metadata, and extracting first relationships between the first entities in the first metadata; Constructing a first knowledge representation corresponding to the first metadata according to the first entity and the first relationship; The first knowledge representation is updated according to the second metadata to obtain a knowledge graph corresponding to the metadata.

3. The method according to claim 2, characterized in that The updating of the first knowledge representation according to the second metadata to obtain a knowledge graph corresponding to the metadata includes: identifying second entities in the second metadata, and extracting second relationships between the second entities in the second metadata; Performing entity alignment on the second entity, wherein the entity alignment includes at least one of entity disambiguation and coreference resolution; Adding the second entity and the second relationship to the first knowledge representation to obtain a second knowledge representation; In the second knowledge representation, a knowledge representation learning model is used to perform entity prediction and relationship prediction to obtain a knowledge graph.

4. The method according to claim 1, characterized in that After extracting entities and the relationships between the entities from the metadata and constructing a knowledge graph corresponding to the metadata according to the entities and the relationships, the method further includes: Associating the knowledge graph with the original data according to the corresponding relationship between the metadata and the original data; A static data knowledge directory corresponding to the metadata is constructed based on the knowledge graph.

5. The method according to claim 1, characterized in that The generating of a dynamic data knowledge catalog in the knowledge graph according to the data retrieval requirement, wherein the dynamic data knowledge catalog is included after the target metadata determined in the metadata according to the data retrieval requirement, and further includes: According to a preset association relationship, acquiring associated metadata corresponding to the target metadata, wherein the preset association relationship includes at least one of a causal association relationship, a correlation association relationship, and an integrated association relationship; A data map is provided to the user according to the target metadata and the associated metadata.

6. The method according to claim 1, characterized in that After collecting and storing metadata corresponding to the original data, the method further includes: The metadata is classified and managed, and the classification management includes at least one of metadata statistics, metadata model definition, metadata classification rule definition, metadata version management, and metadata change management.

7. A data weaving processing device, characterized in that: The device comprises: A collection and storage module, used to collect and store metadata corresponding to the original data, the original data is distributedly stored in more than two data source nodes, and the metadata is used to describe the storage information of the original data in the distributed storage; A knowledge graph module, used to extract entities and relationships between the entities from the metadata, and construct a knowledge graph corresponding to the metadata based on the entities and the relationships; User demand module, used to determine the user's data retrieval needs; A data catalog module, used to generate a dynamic data knowledge catalog in the knowledge graph according to the data retrieval requirement, wherein the dynamic data knowledge catalog includes target metadata determined in the metadata according to the data retrieval requirement, and the dynamic data knowledge catalog refers to a knowledge catalog dynamically generated according to the user's data retrieval requirement and including knowledge representation corresponding to the target metadata; The device also includes: The model application module is used to obtain the corresponding target data and the data source node corresponding to the target data in the original data according to the target metadata, and obtain the corresponding target algorithm according to the data retrieval requirements and the target data, and then decompose the target algorithm into the data source node corresponding to the target data, and perform joint training between the data source nodes based on federated learning to parse the target data, and provide the parsing results of the target data based on the target algorithm.

8. An electronic device, characterized in that: The invention comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the data weaving processing method as claimed in any one of claims 1 to 6.

9. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the data weaving processing method according to any one of claims 1 to 6 is implemented.

Images