Data fabric method and apparatus, and storage medium

Abstract

Provided in the present disclosure are a data fabric method and apparatus, and a storage medium. The method comprises: collecting data from a plurality of data sources, and constructing a knowledge graph according to the data collected from the plurality of data sources, wherein the data collected from the plurality of data sources comprises dynamic metadata and static metadata; determining data to be pushed of a target user on the basis of dynamic metadata of the target user and the knowledge graph; and pushing to the target user the data to be pushed.

Description

Data weaving method, device and storage medium

[0001] This disclosure claims priority to Chinese patent application No. 202310692470.8, filed on June 12, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to the field of data processing technology, and in particular to a data weaving method, device, and storage medium. Background Art

[0003] With the acceleration of enterprise digitalization and the widespread adoption of cloud computing, enterprise data is showing three major trends: continued growth in data volume, diversified data distribution, and the increasing problem of data silos. Analysts predict that enterprise data will increase sixfold over the next three years. Data shelf life is also shortening, and data quickly loses its meaning if not used promptly. Therefore, how to achieve cross-departmental and cross-system enterprise data sharing, enabling internal data analysts to access data at any time and automatically delivering high-quality data to those who need it in real time, while also complying with compliance requirements for data rights and privacy, is the cornerstone and dream of enterprise digital transformation, and also a significant challenge currently facing many enterprises.

[0004] Summary of the Invention

[0005] In one aspect, a data weaving method is provided. The data weaving method comprises:

[0006] Collect data from multiple data sources respectively, and build a knowledge graph based on the data collected from the multiple data sources; wherein the data collected from the multiple data sources include dynamic metadata and static metadata;

[0007] Determine the target user's data to be pushed based on the target user's dynamic metadata and knowledge graph;

[0008] Push the data to be pushed to the target user.

[0009] In another aspect, a data weaving device is provided. The data weaving device comprises:

[0010] The acquisition module is used to collect data from multiple data sources and build a knowledge graph based on the data collected from the multiple data sources; the data collected from the multiple data sources includes dynamic metadata and static metadata;

[0011] A determination module is used to determine the target user's data to be pushed based on the target user's dynamic metadata and knowledge graph;

[0012] The push module is used to push the data to be pushed to the target user.

[0013] In another aspect, a communication node is provided, comprising: a memory and a processor; the memory and the processor are coupled; the memory is configured to store a computer program; and the processor implements the data weaving method described in any one of the above aspects or embodiments when executing the computer program.

[0014] On the other hand, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the data weaving method according to any one of the above aspects or embodiments is implemented.

[0015] On the other hand, a computer program product is provided, which includes computer program instructions, and when the computer program instructions are executed by a processor, the data weaving method according to any one of the above aspects or embodiments is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present disclosure, the following briefly introduces the drawings required for use in some embodiments of the present disclosure. Obviously, the drawings described below are only drawings of some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0017] FIG1 is a schematic diagram of the architecture of a data weaving system according to some embodiments.

[0018] FIG2 is a schematic diagram of a collection module according to some embodiments.

[0019] FIG3 is a schematic diagram of a building block according to some embodiments.

[0020] FIG4 is a schematic diagram of an active metadata module according to some embodiments.

[0021] FIG5 is a schematic diagram of an application module according to some embodiments.

[0022] FIG6 is a flowchart of a data weaving method according to some embodiments.

[0023] FIG7 is a schematic diagram of constructing a knowledge graph according to some embodiments.

[0024] FIG8 is a flowchart of another data weaving method according to some embodiments.

[0025] FIG9 is a schematic structural diagram of a data weaving device according to some embodiments.

[0026] FIG10 is a schematic structural diagram of another data weaving device according to some embodiments. DETAILED DESCRIPTION

[0027] The following will clearly and completely describe the technical solutions of this disclosure in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this disclosure, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of this disclosure without making any creative efforts shall fall within the scope of protection of this disclosure.

[0028] It should be noted that in this disclosure, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this disclosure as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present related concepts in some way.

[0029] In the following, the terms "first," "second," etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the quantity of the technical features indicated. Therefore, a feature specified as "first," "second," etc. may explicitly or implicitly include one or more of the features.

[0030] In the description of this disclosure, unless otherwise specified, " / " means "or." For example, A / B can mean A or B. "And / or" herein is merely a description of an association relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: only A, only B, and both A and B. Furthermore, "at least one" means one or more, and "a plurality" means two or more.

[0031] As mentioned in the background, with the acceleration of enterprise digitalization and the widespread adoption of cloud computing, enterprise data is showing three major trends: a continued increase in data volume, a diversified data distribution, and an increasingly severe data silo problem. Analysts predict that enterprise data will increase sixfold over the next three years. Data shelf life is shortening, and data quickly loses its meaning if not used quickly. Therefore, improving data circulation while ensuring data security, enabling cross-departmental and cross-system enterprise data sharing, and automatically and in real time delivering high-quality data to those in need is the cornerstone and dream of enterprise digital transformation, and also a significant challenge currently facing many enterprises.

[0032] To address the above technical issues, metadata management tools such as metadata-based data catalogs and data maps can be used to organize metadata. Data weaving technology can also be used for data management. Data weaving dynamically coordinates distributed data sources in an intelligent, secure, and self-service manner, providing integrated data to data consumers across data platforms. In other words, regardless of data location, the data weaving architecture enables data consumers to obtain the data they need in real time, accurately, and cost-effectively.

[0033] Data weaving technology primarily focuses on data integration and data virtualization across multi-cloud and hybrid ecosystems, enabling extensive connectivity across various data sources. However, most enterprises only organize metadata without analyzing the relationships between metadata.

[0034] For example, the data management approach may have the following deficiencies:

[0035] (1) Metadata generally only records relatively static metadata information such as the location, function, importance, and how to use the data, and cannot understand the status and usage of the data in the system.

[0036] (2) Metadata management is passive metadata management, that is, it passively accepts metadata input by users or collected metadata, and cannot actively identify and establish data associations.

[0037] (3) Data management tools can only provide search filtering, cannot actively recommend data to users, and cannot meet users' usage needs.

[0038] In response to the above technical problems, the present disclosure proposes a data weaving method, the idea of ​​which is to collect data from multiple data sources, and the collected data includes static metadata and dynamic metadata, which can enrich the business semantics of the data; then, construct a knowledge graph based on the data collected from multiple data sources, which can actively identify and establish data associations, and achieve a more three-dimensional characterization of the data; then, based on the dynamic metadata and knowledge graph of the target user, determine the target user's data to be pushed, and push the data to be pushed to the target user. It can be seen that the data weaving method provided by the embodiment of the present disclosure can actively discover the user's business needs, and recommend business-related data to the user in combination with the knowledge graph, so that the user can obtain the required data in real time and accurately, thereby improving the user experience.

[0039] 1 is an architecture diagram of a data weaving system according to an embodiment of the present disclosure. As shown in FIG1 , the data weaving system 100 includes: a collection module 101 , a construction module 102 , an active metadata module 103 , and an application module 104 .

[0040] The acquisition module 101 is used to acquire data from various data sources.

[0041] In some embodiments, the multiple data sources include, but are not limited to, data lakes, data warehouses, relational databases, cloud storage, and application systems.

[0042] In some embodiments, the raw data from various data sources may include structured data, semi-structured data, or unstructured data. For example, structured data may be data from databases such as enterprise resource planning (ERP), common databases (CDB), and cloud data warehouses (CDW). Semi-structured data may be data such as JSON text and log files. Unstructured data may include documents, images, and videos.

[0043] For unstructured data, the method provided in the embodiment of the present disclosure can interpret the content of unstructured data based on machine learning algorithms to identify metadata from the unstructured data, enrich the business semantics of the metadata, and prepare the content for building a knowledge graph for construction module 102.

[0044] In some embodiments, data collected from various data sources includes metadata, crowdsourced data, and business context information. Exemplarily, metadata includes, but is not limited to, technical metadata, business metadata, operational metadata, social metadata, and localized knowledge (where localized knowledge refers to locally stored, pre-built knowledge graphs). Crowdsourced data includes, but is not limited to, user behavior, user reviews, user ratings, and certifications. Business context information includes, but is not limited to, business terminology, business policies, and data dictionaries.

[0045] In some embodiments, data collected from various data sources includes static metadata and dynamic metadata. Static metadata refers to metadata related to the results of the data, such as user attribute information. Dynamic metadata refers to metadata related to the state and use of the data and is in a continuously changing state, such as data related to user behavior.

[0046] In some embodiments, as shown in FIG2 , the acquisition module 101 includes a collector, an acquisition configuration unit, a receiving and processing unit, a storage adapter unit, and an external interface.

[0047] The collector is used to collect data from the data source according to the information configured by the collection configuration unit.

[0048] For example, as shown in Figure 2, collector types include plugins, scripts, and crawlers. The collector is integrated into the data source and can transmit the collected data to the receiving and processing unit via an interface. For example, the interface between the collector and the receiving and processing unit includes a message interface, a representational state transfer (REST) ​​interface, and an application program interface (API).

[0049] The collection configuration unit is used to interact with the user and supports the user to configure the source, type, model used and other information of the collected data.

[0050] The receiving and processing unit is used to receive the data collected by the collector and perform data analysis. For example, the data analysis includes analyzing data source information, metadata structure information, metadata association information, metadata specification checking, and metadata classification.

[0051] In some embodiments of the present disclosure, for semi-structured data, the receiving and processing unit may first use a wrapper to convert the semi-structured data into structured data; and then perform the above data analysis on the structured data.

[0052] In some embodiments of the present disclosure, the receiving processing unit may not process the unstructured data, but may call the storage adaptation unit to send the unstructured data to the construction module 102, which will extract, fuse, process and store the unstructured data.

[0053] The storage adapter unit is used to store the analyzed and processed metadata in the graph database.

[0054] The construction module 102 is used to construct a knowledge graph based on the data collected by the collection module 101.

[0055] Exemplarily, the construction module 102 may construct a knowledge graph by combining top-down ontology design and bottom-up knowledge extraction.

[0056] In some embodiments, as shown in FIG3 , the process of constructing the knowledge graph by the construction module 102 includes: knowledge modeling, knowledge extraction, knowledge fusion, knowledge processing, etc.

[0057] Knowledge modeling refers to the construction of an ontology. For example, domain business experts and industry experts can collaborate to develop and construct a standardized domain ontology.

[0058] Knowledge extraction refers to extracting entities, attributes and relationships from an ontology.

[0059] In some embodiments, entity parallel relationship similarity calculation and entity hypernym relationship extraction can be performed based on natural language processing (NLP) technology to abstract the ontology from the bottom up.

[0060] For example, knowledge extraction can be performed through an extraction model. First, business experts use corpus annotation tools to manually annotate the corpus according to the specifications of the domain ontology. Simultaneously, with the help of technologies such as remote supervision, semi-automatic corpus annotation is performed as auxiliary corpus annotation to provide data and labels for training the extraction model. Second, based on the annotated corpus and machine learning algorithms, data cleaning, entity extraction, attribute extraction, and relationship extraction are performed to train the extraction model. Based on the trained model, entity extraction, attribute extraction, and relationship extraction from metadata are automatically completed.

[0061] Knowledge fusion refers to the integration of descriptive information about the same entity or concept from multiple data sources. For example, based on the extracted entities, attributes, and relationships, and with the support of an ontology and rule base, entity matching, entity alignment, entity classification, and ontology fusion can be performed to ensure the rigor and attribute consistency of the knowledge graph.

[0062] Knowledge processing can use knowledge reasoning methods to achieve knowledge completion and knowledge error correction, and business experts can evaluate and optimize the quality of the graph.

[0063] After knowledge modeling, knowledge extraction, knowledge fusion, and knowledge processing, the construction module 102 can construct a knowledge graph corresponding to the metadata based on the extracted entities, attributes, and relationships.

[0064] In some embodiments, as shown in FIG3 , the construction module 102 further includes a database, and the construction module 102 is further configured to store the knowledge graph in the database after constructing the knowledge graph corresponding to the metadata.

[0065] In some embodiments of the present disclosure, the database may include a graph database and a relational database; the knowledge graph may be stored in a graph database and / or a relational database. For example, the entire knowledge graph may be stored in a graph database; or, the entire knowledge graph may be stored in a relational database; or, the entities and relationships in the knowledge graph may be stored in a graph database, and the attributes in the knowledge graph may be stored in a relational database.

[0066] A graph database can include graph engine functions and graph storage functions. Graph storage functions refer to the ability to store knowledge graphs, while graph engine functions refer to the ability to query and analyze stored knowledge graphs.

[0067] A relational database may include a variety of data tables for storing data relationships. It is understandable that, since a relational database is in the form of data tables, it cannot store graph topology structures.

[0068] The active metadata module 103 is used to provide personalized metadata recommendation services to users in different business fields and data projects.

[0069] In some embodiments, as shown in FIG4 , the data recommendation process of the active metadata module 103 includes: data recall, data sorting, and data rearrangement.

[0070] In some embodiments, data recall recalls data that the user desires to recall from the raw data of the multiple data sources based on a recall model.

[0071] Recall models include: data collaborative filtering (Iterm-CF) model based on collaborative filtering (CF) algorithm, user collaborative filtering (User-CF) model, graph embedding model, multi-interest network with dynamic routing (MIND) model, crowd-based filtering model, etc.

[0072] In some embodiments, data sorting is based on a sorting model for determining a rank for the recall data.

[0073] Sorting models include: feature engineering model, logistic regression (LR) model, behavior sequence transformer (BST) model, deep factorization machine (Deep FM) model, deep residual network (DeepResNet) model, etc.

[0074] In some embodiments, the data shuffling adjusts the ranking of the recall data based on the shuffling model.

[0075] The reordering model is built based on manually set rules; alternatively, the reordering model is built based on machine learning strategies. It is understandable that the reordering model needs to be built and trained based on human experience and human understanding of the business.

[0076] In some embodiments of the present disclosure, the data rearrangement mechanism may include at least one of the following: rearrange the data after deduplication and fragmentation, rearrange the data according to the diversity of the data, rearrange the data after multi-task fusion, rearrange the data after improving the real-time performance of the rearrangement model, and rearrange the data according to rule strategies, etc.

[0077] For example, data rearrangement involves breaking up the recommended data into multiple dimensions as needed, improving the real-time performance of the rearrangement model (improving the real-time performance of the rearrangement model involves quantizing and pruning the model to optimize its performance and efficiency), and obtaining the final data recommendation results. For example, a standard category system can be established in conjunction with the knowledge graph to expand data categories at the semantic level.

[0078] It should be noted that data rearrangement is an optional step that can be designed specifically according to business needs or incorporated into the data sorting step.

[0079] The application module 104 is used to recommend data to users of the third-party service platform according to the needs of the target users of the third-party service platform.

[0080] In some embodiments, as shown in FIG5 , the data weaving system 100 provided by the embodiment of the present disclosure can be integrated with a third-party business platform to obtain the needs of the target users of the third-party business platform, determine the target users' data to be recommended based on the needs of the target users of the third-party business platform, and recommend the data to be recommended to the target users.

[0081] As a possible implementation, third-party business platforms can be integrated into the data weaving system 100. In this way, users can complete data services such as service sharing, business intelligence (BI) reporting, and data quality management by operating a single context in the data weaving system 100. Furthermore, the data weaving system 100 can provide third-party business platforms with metadata search, recommendation, classification, and lineage functions.

[0082] As another possible implementation, the data weaving system 100 can be integrated into a third-party business platform. This allows users to search for metadata and view personalized metadata recommendations within the third-party business platform. Furthermore, by clicking on the retrieved metadata or personalized metadata recommendations, users can be redirected to the corresponding metadata interface to view and request data.

[0083] In some embodiments of the present disclosure, the data weaving system provided by the embodiments of the present disclosure can be integrated with a third-party business platform through various technologies such as Java database connectivity (JDBC), REST, API interface, and messaging.

[0084] In some embodiments, the application module 104 includes: a service configuration unit, a service management unit, a rule management unit, and a recommendation implementation unit.

[0085] The service configuration unit is used to visually display the basic configuration information embedded in the third-party business platform to users, including the third-party business platform's application address, icon, login, authentication, and other information. The service configuration unit also supports single sign-on and permission management for connecting to the third-party business platform.

[0086] It is understandable that the integration of third-party business platforms solves the problem of business silos. The third-party business platform can not only provide metadata to the data weaving system 100, but also use the metadata and proactive recommendation functions of the data weaving system 100. In this way, users can find and use data in a single context.

[0087] The service management unit is used to manage third-party applications that have been integrated into the system and set access permissions for users of the data weaving system to third-party business platforms. For example, through the service management unit, administrators can view the third-party business platforms that have been integrated into the data weaving system and configure different access permissions for different users.

[0088] The rule management unit is used to obtain the smart services selected by the third-party business platform and their related configurations, completing the initialization of the smart service environment. For example, smart service-related configurations include, but are not limited to, configuring the service type, service parameters, service implementation interface, and third-party information collection. The rule management unit calculates and applies for resources based on the parameters configured by the third-party business platform.

[0089] Exemplarily, the service types of the third-party business platform may include at least one of the following: a data analysis and recommendation platform, a security and compliance management platform, an intelligent collection and orchestration platform, and an intelligent operation and maintenance recommendation platform.

[0090] The data analysis recommendation platform can proactively recommend data to users based on their data needs, reducing the user's workload in data discovery. The security and compliance management platform can intelligently classify and grade data, and then protect the data based on the results of classification and grading. For example, based on the results of data classification and grading, manage user permissions; or, based on the results of data classification and grading, encrypt data; or, based on the results of data classification and grading, desensitize data (for example, perform operations such as obfuscating, replacing, clearing, and encrypting sensitive data). The intelligent collection and orchestration platform can be used to recommend data pipeline orchestration to users and automatically and dynamically allocate resources. The intelligent operation and maintenance recommendation platform can be used to proactively prevent network alarms and user complaints and identify equipment risks.

[0091] The recommendation implementation unit is used to obtain information from third-party business platforms in real time (e.g., request information, model information, function information, access control information, etc.), and then trigger the active metadata module 103 through triggers to provide personalized data recommendation services to users. For example, personalized data recommendation services include but are not limited to: data analysis recommendations, security and compliance management recommendations, intelligent collection and orchestration recommendations, and intelligent operation and maintenance recommendations.

[0092] In some embodiments of the present disclosure, the recommended implementation unit serves as an adaptation layer of the data weaving system 100, which can shield the differences in intelligent data service requirements of different third-party business platforms, including but not limited to: differences in input interface types, format differences, and differences in output result processing methods.

[0093] In some embodiments of the present disclosure, the data weaving system 100 can provide metadata query services to third-party business platforms. For example, third-party business systems can query metadata through various methods, such as Kafka, REST, and APIs. Data query content includes, but is not limited to, data, users, business categories, data quality, and data statistics.

[0094] It should be noted that the system architecture and application scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0095] The following is an exemplary introduction to a data weaving method provided by an embodiment of the present disclosure.

[0096] For example, as shown in FIG6 , a data weaving method provided by an embodiment of the present disclosure can be implemented as follows:

[0097] S201. Collect data from multiple data sources respectively.

[0098] The data collected from various data sources include: dynamic metadata and static metadata.

[0099] In some embodiments, the data collected from multiple data sources also includes at least one of the following: metadata representing data source information, metadata representing business attributes, user behavior data, user preference data, user attribute data, and business context information.

[0100] Metadata representing data source information is also called technical metadata. For example, technical metadata may include data source information, database table and field names, field attributes, physical data models, processing details (such as the extraction, cleansing, transformation, and loading (ETL) process), and data lineage documentation (including information on the impact of upstream and downstream changes, data access permissions, groups, and roles).

[0101] Metadata that characterizes business attributes is also referred to as business metadata. For example, business metadata may include: business definitions of data sets, explanations of business terminology, business metric names, calculation methods, derived metrics, business rule engine rules, data quality detection rules, data mining algorithms, and data security or sensitivity levels.

[0102] User behavior data may include: number of visits / applications, page browsing time, visit duration, page dwell time, browsing path, etc.

[0103] User preference data may include: login method, browsed content, commented content, interactive content, favorite content, data preferences, etc.

[0104] User attribute data can also be referred to as user basic information data. For example, user basic information data may include: name, gender, date of birth, place of origin, job title, department, authority, active time, etc.

[0105] Business context information may include business terms, business policies, and data dictionaries.

[0106] In some embodiments of the present disclosure, data collected from various data sources may also include operational metadata. For example, operational metadata may include: data access method, access time, access restrictions, data access permissions, groups and roles, data processing job results, system execution logs, data backups, archivers, and archive time.

[0107] In some embodiments of the present disclosure, data collected from various data sources may also include content enhancement metadata. For example, this content enhancement metadata may include electronic document data extracted through content interpretation of unstructured data using machine learning algorithms. For example, content enhancement metadata may include keywords, tags, summaries, semantic relevance, and text similarity in sales contracts.

[0108] As a possible implementation method, the above step S201 can be implemented as follows: collecting technical metadata from database data sources such as MySQL, Oracle, Elasticsearch of the business system through Kafka, REST, API, etc.; collecting business metadata, management metadata, social metadata, crowdsourcing metadata, etc. from BI tools, ETL tools, governance tools, applications (Application, APP), etc.

[0109] It is understood that the method provided by the embodiments of the present disclosure adopts an open metadata collection framework and sets up a large number of metadata collectors for the data ecosystem. In this way, new data sources can be quickly and cheaply accessed to form data context information that completes the link (for example, real-time, detailed, and accurate full-link lineage).

[0110] In some embodiments, the raw data in the various data sources include structured data, semi-structured data, and unstructured data.

[0111] In some embodiments, the multiple data sources include a third-party business platform. Step S201 may be implemented as follows: collecting data of a target user from the third-party business platform. The target user is a user of the third-party business platform. The data of the target user includes dynamic metadata of the target user.

[0112] S202. Construct a knowledge graph based on data collected from multiple data sources.

[0113] In some embodiments, the above step S202 can be implemented as the following steps:

[0114] Step a1: Construct an ontology based on data collected from multiple data sources.

[0115] For example, a standardized domain ontology can be developed collaboratively by domain business experts and industry experts.

[0116] Step a2: Extract entities, attributes and relationships from the ontology.

[0117] In some embodiments, entity parallel relationship similarity calculation and entity hierarchical relationship extraction can be performed based on NLP technology to abstract the ontology from the bottom up.

[0118] Exemplarily, knowledge extraction can be performed through an extraction model. First, business experts use corpus annotation tools to complete manual corpus annotation under the specifications of the domain ontology. At the same time, with the help of technologies such as remote supervision, semi-automatic corpus annotation is completed as auxiliary corpus annotation to provide data and labels for training the extraction model. Secondly, based on the annotated corpus and machine learning algorithms, extraction model training for data cleaning, entity extraction, attribute extraction and relationship extraction is performed. Exemplarily, the basic model for entity extraction can be a bidirectional Transformer encoder (bidirectional encoder representations from transformers, BERT), a bidirectional long short-term memory network (bi-directional long short-term memory, BiLSTM), and a conditional random field (conditional random fields, CRF). Based on the trained model, entity extraction, attribute extraction and relationship extraction of metadata are automatically completed.

[0119] In some embodiments, the above method may further include: based on the extracted entities, attributes, and relationships, with the support of the ontology library and rule library, completing entity matching, entity alignment, entity classification, and ontology fusion. In this way, the rigor and attribute consistency of the knowledge graph can be ensured.

[0120] In some embodiments, the above method may further include: using knowledge reasoning methods to achieve knowledge completion and knowledge error correction, and having business experts evaluate and optimize the quality of the graph.

[0121] Step a3: Build a knowledge graph based on the extracted entities, attributes, and relationships.

[0122] It is understood that the knowledge graph construction method provided by the embodiments of this disclosure is based on static and dynamic metadata collected from multiple data sources, constructing metadata knowledge graphs in a top-down and bottom-up manner, and quantifying entity connections and connection relationships based on artificial intelligence or machine learning algorithms. In this way, it is possible to automatically mine and establish associations between metadata, forming a semantic knowledge graph, thereby achieving a more three-dimensional portrayal of data and supporting more intelligent data usage recommendations.

[0123] To facilitate understanding, the knowledge graph construction process provided by the embodiment of the present disclosure is described below in the form of examples.

[0124] For example, as shown in Figure 7, the process of building a knowledge graph is explained by taking the data processing pipeline as an example:

[0125] First, domain experts build an ontology based on business analysis, such as data sources, data sets, processing procedures, users, and user groups. The input and output of the processing procedures are data sets, the owner of the data set is the user, and the user is a member of the user group.

[0126] Secondly, entities, relationships, and attributes are extracted from the collected metadata based on ontology and machine learning algorithms designed by domain experts. For example, the entities, relationships, and attributes of process 1 shown in Figure 7 are extracted based on the collected algorithm content, including the structured query language (SQL) corresponding to the process, the data source of the process, the input dataset, the output dataset, and the lineage relationship.

[0127] Next, the extracted knowledge is integrated with the existing knowledge. For example, the processing process 1 shown in FIG7 is used to perform entity matching and user clustering with dataset 1 and dataset 2 to complete the ontology fusion.

[0128] Finally, the metadata knowledge graph constructed based on the above entities, attributes and relationships is saved to the graph database after expert quality assessment and optimization.

[0129] S203: Determine the target user's data to be pushed based on the target user's dynamic metadata and knowledge graph.

[0130] Exemplarily, the dynamic metadata of the target user may be the access behavior metadata of the target user.

[0131] In some embodiments, as shown in FIG8 , the above step S203 can be implemented as the following steps:

[0132] S2031. Input the target user's dynamic metadata and knowledge graph into M recall models to obtain at least one recall data.

[0133] The recall model is used to recall the data that the user expects to recall from the original data of multiple data sources; M is an integer greater than or equal to 1.

[0134] For example, a recall strategy based on a graph embedding model can be implemented as follows: based on user access behavior metadata, a graph embedding model is used to generate data embedding vectors; based on the data embedding vectors, similarities between data are calculated; and then, based on the similarities between data, recall data is determined. For example, the following steps can be implemented:

[0135] Step b1: Construct a directed graph of data based on user access behavior metadata.

[0136] Step b2: Select a starting point from the directed graph, and then use a deep random walk algorithm to generate a data sequence.

[0137] Step b3: Introduce supplementary information (side information) of the data based on the knowledge graph.

[0138] Exemplarily, the supplementary information of the data includes: content enhancement metadata such as category, description, keywords, etc. to enrich the representation, and train the Word2vec model to generate the embedding vector representation of the data.

[0139] Step b4: Calculate the similarity score between the two data based on the embedding vector representation and generate an i2i table.

[0140] Step b5: query the i2i table through user triggers to retrieve candidate recommendation data.

[0141] It should be noted that the embodiments of the present disclosure provide a variety of recall strategies, and different recall strategies can be selected according to different scenarios. For example, a single-channel recall strategy of a recall model is selected. Alternatively, multiple recall models are selected for multi-channel recall, and a multi-channel recall strategy that fuses the multi-channel recall data (it is understandable that the multi-channel recall strategy can improve the accuracy of the recall data). Alternatively, a recall strategy of a custom recall model is adopted.

[0142] For example, the method provided by the embodiments of the present disclosure can group users with similar observation patterns together to recommend similar content. The similarity between different users is calculated based on the user's behavioral metadata, and then the user groups are clustered according to preset thresholds or preset rules. Based on the historical behavioral metadata of each user group, the data with a high access or application rate of the user group is recalled as candidate recommendation data for the user group. By manually setting empirical parameters and rules, the data recalled based on the graph embedding algorithm and the data recalled based on user clustering are fused to form the final candidate recommendation data.

[0143] S2032: Determine the data to be pushed to the target user according to at least one recalled data.

[0144] As a possible implementation manner, the above step S2032 may be implemented as: using at least one recalled data as the data to be pushed to the target user.

[0145] As another possible implementation, the data collected from the multiple data sources includes the target user's data. The above step S2032 can be implemented as the following steps:

[0146] Step c1: extract the target user's features from the target user's data.

[0147] The characteristics of target users include their behavioral characteristics, attribute characteristics, and preference characteristics.

[0148] Step c2: Input the target user's features and at least one recalled data into N ranking models to obtain a ranking of at least one recalled data.

[0149] The ranking model is used to determine the degree of match between each recalled data and the characteristics of the target user, and to determine the ranking of the recalled data based on the degree of match.

[0150] Step c3: Determine the data to be pushed to the target user from the at least one recalled data according to the ranking of the at least one recalled data.

[0151] For example, taking the Deep FM model as an example, the sorting strategy based on the Deep FM model is described:

[0152] Step d1: extract user features based on the collected data set.

[0153] Exemplarily, user features include: user attribute features, user behavior features, user preference features, data-side features, interaction features, contextual features, and other features. User attribute features may include metadata of basic user features, such as the user's level, position, and department. User behavior features may include the data sequence accessed by the user, the data sequence queried by the user, the access duration, and the corresponding position deviation. User preference features may include data reflecting user preferences, such as user access duration data, user search content data, and user click content data. Data-side features may include: metadata of basic data features, such as data identity (ID), number of visits, source, and creation time; and content enhancement metadata, such as keywords, categories, and themes of the data. Other features such as interaction features and contextual features can be extracted according to the needs of the application field.

[0154] As a possible implementation, user features can be extracted based on a feature extraction model. For example, the collected data set is input into a trained feature extraction model to obtain user features.

[0155] It should be noted that the embodiments of this disclosure provide a variety of feature engineering algorithms, including but not limited to feature extraction, dimensionality reduction, and processing. In the embodiments of this disclosure, you can select an appropriate feature extraction algorithm or import a custom feature engineering model based on different scenarios. Alternatively, you can combine multiple feature extraction algorithms to extract user features, and then perform feature fusion based on a fusion strategy to obtain the final user features.

[0156] Step d2: Train the Deep FM model based on user characteristics.

[0157] Exemplarily, user-side features, user behavior features, data-side features, interaction features, context features, and other features are processed and fused, and then input into the Deep FM model together with the label (0 indicates no visit, 1 indicates visit) for training.

[0158] Step d3: Based on the trained Deep FM model, predict the user's next click data.

[0159] Step d4: Determine the ranking of the recalled data based on the prediction results of the Deep FM model.

[0160] It should be noted that the disclosed framework provides a variety of feature engineering algorithms, including but not limited to feature extraction, dimensionality reduction, and processing. Based on different scenarios, you can select the appropriate feature extraction algorithm or import a custom feature engineering model. You can also combine multiple feature extraction algorithms and perform feature fusion through manually designed fusion strategies. Similarly, the disclosed framework provides a variety of ranking models. You can select different ranking models based on different scenarios or import custom ranking models.

[0161] It should be noted that the embodiments of the present disclosure provide a variety of sorting models, and different sorting models can be selected according to different scenarios or custom sorting models can be imported.

[0162] In some embodiments, the above step c3 can be implemented as the following steps:

[0163] Step c3-1: Determine at least one candidate data from the at least one recalled data according to the ranking of the at least one recalled data.

[0164] The at least one candidate data is data in the at least one recalled data whose ranking is greater than a first preset threshold. For example, assuming that there are 500 at least one recalled data, the top 100 recalled data in the at least one recalled data can be used as the at least one candidate data.

[0165] Step c3-2: perform data processing on at least one candidate data to obtain at least one processed candidate data; wherein the data processing includes at least one of the following: fragmentation, deduplication, fusion (such as multi-task fusion), diversity extraction, real-time improvement, etc.

[0166] Step c3-3: Adjust the ranking of at least one candidate data according to the processed at least one candidate data.

[0167] Step c3-4: Determine the data to be pushed to the target user from the at least one candidate data based on the adjusted ranking of the at least one candidate data. Exemplarily, the candidate data in the at least one candidate data whose ranking is greater than a second preset threshold after adjustment is determined as the data to be pushed to the target user. For example, assuming there are 100 at least one candidate data, the top ten candidate data in the at least one candidate data after adjustment may be used as the data to be pushed to the target user.

[0168] It is understandable that, because the data weaving method provided by the disclosed embodiments can be applied to different third-party business platforms, the disclosed embodiments support selecting different data processing methods to process at least one piece of recalled data based on different business needs; and then adjusting the ranking of at least one piece of recalled data based on the processed at least one piece of recalled data. In this way, the data to be pushed to the target user determined based on the ranking of the at least one piece of recalled data better meets the target user's business needs, thereby improving the accuracy of the recommendation results.

[0169] S204: Push the data to be pushed to the target user.

[0170] In some embodiments, the above step S204 may be implemented as: sending the data to be pushed to a third-party service platform, so that the third-party service platform pushes the data to be pushed to the target user.

[0171] For example, the data weaving system can send the data to be pushed to the third-party business platform through an interface. For example, the interface can be JDBC, REST, API interface or message.

[0172] It can be understood that the method provided by the embodiment of the present disclosure can collect data from multiple data sources (wherein the multiple data sources include third-party business platforms, and the data collected from the multiple data sources include the data of the target user), and the collected data includes static metadata and dynamic metadata, which can enrich the business semantics of the data; furthermore, a knowledge graph is constructed based on the data collected from the multiple data sources, which can actively identify and establish data associations, and achieve a more three-dimensional characterization of the data; then, based on the dynamic metadata and knowledge graph of the target user, the data to be pushed of the target user is determined, and the data to be pushed is pushed to the target user. It can be seen that the data weaving method provided by the embodiment of the present disclosure can actively discover the business needs of users, and recommend business-related data to users in combination with the knowledge graph, so that users can obtain the required data in real time and accurately, thereby improving the user experience.

[0173] In some embodiments, the third-party business platform includes at least one of the following: a data analysis recommendation platform, a security and compliance management platform, an intelligent collection and orchestration platform, and an intelligent operation and maintenance recommendation platform.

[0174] For example, when the third-party business platform is a data analysis recommendation platform, taking smart asset recommendation as an example, the method provided by the embodiment of the present disclosure can combine metadata collected from multiple data sources to characterize the relationship between users, data and businesses, and recommend the data set that may be of most interest to each user, thereby greatly improving data discovery efficiency.

[0175] For example, when the third-party business platform is a security and compliance management platform, taking intelligent data classification as an example, the method provided by the embodiment of the present disclosure can realize automatic identification of sensitive information based on metadata and field content sampling collected from multiple data sources, complete accurate data classification and grading, and combine security policies to conduct security and compliance management of data access, sharing and transmission processes.

[0176] For example, when the third-party business platform is an intelligent collection and orchestration platform, taking intelligent operation and maintenance recommendation as an example, the method provided by the embodiment of the present disclosure can proactively identify equipment hidden dangers based on metadata collected from multiple data sources, predict user complaints and network traffic, etc., and thus proactively prevent network alarms and user complaints.

[0177] For example, when the third-party business platform is an intelligent operation and maintenance recommendation platform, taking intelligent collection and orchestration as an example, the method provided by the embodiment of the present disclosure can recommend data pipeline orchestration methods to users based on metadata collected from multiple data sources, automatically allocate resources, and quickly support current and future needs.

[0178] It is understandable that, in order to realize the above functions, the data weaving device includes hardware structures and / or software modules corresponding to the execution of each function. It should be easy for those skilled in the art to realize that, in combination with the algorithm steps of each example described in the embodiments of the present disclosure, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present disclosure.

[0179] The embodiment of the present disclosure can divide the data weaving device into functional modules according to the above-mentioned method embodiment. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one functional module. The above-mentioned integrated module can be implemented in the form of hardware or software. It should be noted that the division of modules in the embodiment of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation. The following is an example of dividing each functional module corresponding to each function.

[0180] FIG9 is a schematic diagram of the structure of a data weaving device provided by an embodiment of the present disclosure, which can execute the data weaving method provided by the above method embodiment. As shown in FIG9 , the data weaving device 300 includes: a collection module 301 , a determination module 302 and a push module 303 .

[0181] The collection module 301 is used to collect data from multiple data sources respectively and construct a knowledge graph based on the data collected from the multiple data sources; wherein the data collected from the multiple data sources include dynamic metadata and static metadata.

[0182] The determination module 302 is used to determine the data to be pushed of the target user based on the dynamic metadata and knowledge graph of the target user.

[0183] The push module 303 is used to push the data to be pushed to the target user.

[0184] In some embodiments, the determination module 302 can be used to input the dynamic metadata and knowledge graph of the target user into M recall models to obtain at least one recall data; wherein the recall model is used to recall the data that the user expects to recall from the original data of multiple data sources; M is an integer greater than or equal to 1; and the data to be pushed to the target user is determined based on the at least one recall data.

[0185] In some embodiments, the data collected from multiple data sources includes the data of the target user; the determination module 302 can be used to extract the characteristics of the target user from the target user's data; wherein, the characteristics of the target user include the target user's behavioral characteristics, attribute characteristics, and preference characteristics; the characteristics of the target user and at least one recalled data are input into N sorting models to obtain the ranking of at least one recalled data; the sorting model is used to determine the degree of match between each recalled data and the characteristics of the target user, and determine the ranking of the recalled data based on the degree of match; based on the ranking of at least one recalled data, the data to be pushed for the target user is determined from the at least one recalled data.

[0186] In some embodiments, the determination module 302 can be used to determine at least one candidate data from at least one recalled data based on the ranking of at least one recalled data; the at least one candidate data is data whose ranking in at least one recalled data is greater than a preset threshold; data processing is performed on the at least one candidate data to obtain at least one processed candidate data; the data processing includes at least one of the following: breaking up, deduplication, and fusion; adjusting the ranking of at least one candidate data based on the processed at least one candidate data; determining the data to be pushed to the target user from the at least one candidate data based on the adjusted ranking of the at least one candidate data.

[0187] In some embodiments, the target user is a user of a third-party business platform; the collection module 301 is used to collect data of the target user from the third-party business platform; the data of the target user includes dynamic metadata of the target user; the push module 303 is used to send the data to be pushed to the third-party business platform, so that the third-party business platform pushes the data to be pushed to the target user.

[0188] In some embodiments, the third-party business platform includes at least one of the following: a data analysis recommendation platform, a security and compliance management platform, an intelligent collection and orchestration platform, and an intelligent operation and maintenance recommendation platform.

[0189] In some embodiments, the data collected from multiple data sources also includes at least one of the following: metadata representing data source information, metadata representing business attributes, user behavior data, user preference data, user attribute data, and business context information.

[0190] In some embodiments, the raw data in the various data sources include structured data, semi-structured data, and unstructured data.

[0191] In the case of implementing the functions of the above-mentioned integrated modules in hardware, the embodiments of the present disclosure provide another possible structure of the data braiding device involved in the above-mentioned embodiments. As shown in Figure 10, the data braiding device 400 includes: a processor 402 and a bus 404. In some embodiments of the present disclosure, the data braiding device 400 may also include a memory 401; in some embodiments of the present disclosure, the data braiding device 400 may also include a communication interface 403.

[0192] Processor 402 can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of this disclosure. Processor 402 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof, and can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of this disclosure. Processor 402 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0193] The communication interface 403 is used to connect to other devices via a communication network, such as Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0194] The memory 401 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0195] As a possible implementation, memory 401 can exist independently of processor 402 and can be connected to processor 402 via bus 404 to store instructions or program codes. When processor 402 calls and executes the instructions or program codes stored in memory 401, the data weaving method provided in the embodiment of the present disclosure can be implemented.

[0196] In another possible implementation, the memory 401 may also be integrated with the processor 402 .

[0197] Bus 404 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 404 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, FIG10 shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0198] Some embodiments of the present disclosure provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium), which stores computer program instructions. When the computer program instructions are executed on a computer, the computer executes the data weaving method described in any of the above embodiments.

[0199] Exemplarily, the above-mentioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes, etc.), optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memories (EPROMs), cards, sticks, or key drives, etc.). The various computer-readable storage media described in the present disclosure may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0200] An embodiment of the present disclosure provides a computer program product comprising instructions. When the computer program product is run on a computer, the computer is enabled to execute the data weaving method described in any one of the above embodiments.

[0201] The above is only a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or replacements within the technical scope disclosed in the present disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A data weaving method, comprising: Collecting data from a plurality of data sources respectively, and constructing a knowledge graph according to the data collected from the plurality of data sources; wherein the data collected from the plurality of data sources includes dynamic metadata and static metadata; Determine the data to be pushed for the target user based on the dynamic metadata of the target user and the knowledge graph; Push the data to be pushed to the target user.

2. The method according to claim 1, wherein: Based on the dynamic metadata of the target user and the knowledge graph, determining the data to be pushed of the target user includes: Input the dynamic metadata of the target user and the knowledge graph into M recall models to obtain at least one recall data; wherein the recall model is used to recall the data that the user expects to recall from the original data of the multiple data sources; and M is an integer greater than or equal to 1; The data to be pushed to the target user is determined according to the at least one recalled data.

3. The method according to claim 2, wherein: The data collected from the multiple data sources include the data of the target user; and the step of determining the data to be pushed to the target user according to the at least one recalled data includes: Extracting the characteristics of the target user from the data of the target user; wherein the characteristics of the target user include the behavior characteristics, attribute characteristics, and preference characteristics of the target user; Inputting the target user's characteristics and the at least one recalled data into N ranking models to obtain the ranking of the at least one recalled data; the ranking model is used to determine the matching degree between each recalled data in the at least one recalled data and the target user's characteristics, and determine the ranking of each recalled data according to the matching degree; N is an integer greater than or equal to 1; According to the ranking of the at least one recalled data, the data to be pushed to the target user is determined from the at least one recalled data.

4. The method according to claim 3, wherein: The step of determining the to-be-pushed data of the target user from the at least one recalled data according to the ranking of the at least one recalled data comprises: Determining at least one candidate data from the at least one recalled data according to the ranking of the at least one recalled data; the at least one candidate data is data whose ranking in the at least one recalled data is greater than a preset threshold; Performing data processing on the at least one candidate data to obtain the at least one processed candidate data; the data processing includes at least one of the following: breaking up, deduplication, and fusion; Adjusting the ranking of the at least one candidate data according to the processed at least one candidate data; According to the adjusted ranking of the at least one candidate data, the data to be pushed to the target user is determined from the at least one candidate data.

5. The method according to claim 1, wherein: The target users are users of a third-party business platform; The data are collected from a variety of data sources, including: Collecting the target user's data from the third-party service platform; the target user's data includes the target user's dynamic metadata; The step of pushing the data to be pushed to the target user includes: The data to be pushed is sent to the third-party service platform, so that the third-party service platform pushes the data to be pushed to the target user.

6. The method according to claim 5, wherein: The third-party business platform includes at least one of the following: a data analysis recommendation platform, a security and compliance management platform, an intelligent collection and orchestration platform, and an intelligent operation and maintenance recommendation platform.

7. The method according to claim 1, wherein: The data collected from the multiple data sources also include at least one of the following: Metadata representing data source information, metadata representing business attributes, user behavior data, user preference data, user attribute data, and business context information.

8. The method according to claim 1, wherein: The original data in the multiple data sources include: structured data, semi-structured data and unstructured data.

9. A communication device, comprising: a processor and a memory for storing instructions executable by the processor; The processor is configured to execute the instructions so that the communication device performs the data weaving method according to any one of claims 1 to 8.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the data weaving method according to any one of claims 1 to 8.

Images