Data weaving method and device and storage medium

Abstract

The invention provides a data weaving method and device and a storage medium, relates to the technical field of data processing, and aims to actively discover service requirements of a user and recommend service associated data to the user in combination with a knowledge graph, so that the user can accurately obtain the required data in real time. The method comprises the following steps: respectively acquiring data from various data sources, and constructing a knowledge graph according to the data acquired from the various data sources; wherein the data collected from the multiple data sources comprises dynamic metadata and static metadata; determining to-be-pushed data of the target user based on the dynamic metadata of the target user and the knowledge graph; and pushing the to-be-pushed data to the target user.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data weaving method, device and storage medium. Background Art

[0002] With the acceleration of enterprise digitalization and the popularization of cloud computing, enterprise data presents three major trends: the trend of continuous growth in data volume, the trend of diversified data distribution, and the increasingly severe problem of data islands. Analysis agencies predict that enterprise data will increase sixfold in the next three years; the shelf life of data is shortened, and if data is not used quickly, it will soon lose its meaning. Therefore, how to achieve cross-departmental and cross-system enterprise data sharing, so that internal data analysis experts can obtain data at any time, and automatically and in real time push high-quality data to those in need, while meeting compliance requirements in terms of data permissions and privacy, is the cornerstone and dream of enterprises to achieve digital transformation, and it is also a huge challenge currently faced by many enterprises.

[0003] Related technologies generally use data management tools such as data catalogs and data maps based on metadata. However, the data management tools in related technologies can only provide search filtering, but cannot actively recommend data to users, and cannot meet the user's usage needs. Summary of the invention

[0004] The embodiments of the present application provide a data weaving method, device and storage medium, which can actively discover the business needs of users and recommend business-related data to users in combination with knowledge graphs, so that users can obtain the required data in real time and accurately.

[0005] It can analyze user needs based on user data and automatically push data to data demanders.

[0006] In one aspect, a data weaving method is provided, comprising:

[0007] Collect data from multiple data sources respectively, and build a knowledge graph based on the data collected from the multiple data sources; wherein the data collected from the multiple data sources include dynamic metadata and static metadata;

[0008] Determine the target user's data to be pushed based on the target user's dynamic metadata and knowledge graph;

[0009] Push the data to be pushed to the target users.

[0010] On the other hand, a data knitting device is provided, comprising:

[0011] A collection module, used to collect data from various data sources respectively, and construct a knowledge graph based on the data collected from the various data sources; wherein the data collected from the various data sources include dynamic metadata and static metadata;

[0012] A determination module is used to determine the target user's data to be pushed based on the target user's dynamic metadata and knowledge graph;

[0013] The push module is used to push the data to be pushed to the target users.

[0014] On the other hand, a communication node is provided, comprising: a memory and a processor; the memory and the processor are coupled; the memory is used to store a computer program; and the processor implements the data weaving method of any of the above embodiments when executing the computer program.

[0015] On the other hand, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the data weaving method of any of the above embodiments is implemented.

[0016] On the other hand, a computer program product is provided. The computer program product includes computer program instructions. When the computer program instructions are executed by a processor, the data weaving method of any of the above embodiments is implemented.

[0017] The data weaving method provided in the embodiment of the present application can collect data from a variety of data sources, and the collected data includes static metadata and dynamic metadata, which can enrich the business semantics of the data; further, a knowledge graph is constructed based on the data collected from a variety of data sources, which can actively identify and establish data associations, and achieve a more three-dimensional characterization of the data; then, based on the dynamic metadata and knowledge graph of the target user, the data to be pushed of the target user is determined, and the data to be pushed is pushed to the target user. It can be seen that the data weaving method provided in the embodiment of the present application can actively discover the business needs of users, and recommend business-related data to users in combination with the knowledge graph, so that users can obtain the required data in real time and accurately, improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in some embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only drawings of some embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0019] Figure 1 A schematic diagram of the architecture of a data weaving system provided for some embodiments of the present application;

[0020] Figure 2 A schematic diagram of a collection module provided in some embodiments of the present application;

[0021] Figure 3 A schematic diagram of a building block provided for some embodiments of the present application;

[0022] Figure 4 A schematic diagram of an active metadata module provided for some embodiments of the present application;

[0023] Figure 5 A schematic diagram of an application module provided for some embodiments of the present application;

[0024] Figure 6 A process of a data weaving method provided for some embodiments of the present application Figure 1 ;

[0025] Figure 7 A schematic diagram of constructing a knowledge graph provided for some embodiments of the present application;

[0026] Figure 8 A process of a data weaving method provided for some embodiments of the present application Figure 2 ;

[0027] Fig. 9 A schematic diagram of a data weaving device provided in some embodiments of the present application Figure 1 ;

[0028] Fig.10 A schematic diagram of a data weaving device provided in some embodiments of the present application Figure 2 . DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0031] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0032] In the description of this application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" means one or more, and "multiple" means two or more.

[0033] As the background technology, with the acceleration of the enterprise digitalization process and the popularization of cloud computing, enterprise data shows three major trends: the trend of continuous growth in data volume, the trend of diversified data distribution, and the increasingly severe problem of data islands. Analysis agencies predict that enterprise data will increase sixfold in the next three years; the shelf life of data will be shortened, and if data is not used quickly, it will soon lose its meaning. Therefore, how to improve the circulation of data while ensuring data security, realize the sharing of enterprise data across departments and systems, and automatically and in real time push high-quality data to those in need is the cornerstone and dream of enterprises to achieve digital transformation, and it is also a huge challenge currently faced by many enterprises.

[0034] In response to the above technical problems, the relevant technology can use metadata-based data catalogs, data maps and other data management tools to comb metadata. The relevant technology also proposes a way to use data weaving technology for data management. Among them, data weaving is to dynamically coordinate distributed data sources in an intelligent, secure and self-service manner, and then provide integrated data to data consumers across data platforms. In other words, no matter where the data is, through the data architecture of data weaving, it can help data consumers obtain the required data in real time, accurately and at low cost.

[0035] Currently, data weaving technology mainly focuses on data integration and data virtualization across multi-cloud and hybrid ecosystems, and is used to achieve extensive connections between various data sources. However, most companies only sort out the metadata of the data, and do not analyze the relationship between the metadata.

[0036] Specifically, the data management method in the related art has the following deficiencies:

[0037] (1) The metadata in related technologies generally only records relatively static metadata information such as the location, function, importance, and how to use the data, but cannot understand the status and usage of the data in the system.

[0038] (2) The metadata management in the related art is passive metadata management, that is, it passively accepts metadata input by users or collected metadata, and cannot actively identify and establish data associations.

[0039] (3) The data management tools in the related technology can only provide search filtering, but cannot actively recommend data to users and cannot meet the user's usage needs.

[0040] In response to the above technical problems, this application proposes a data weaving method, the idea of ​​which is: to collect data from multiple data sources, and the collected data includes static metadata and dynamic metadata, which can enrich the business semantics of the data; then, to construct a knowledge graph based on the data collected from multiple data sources, which can actively identify and establish data associations, and achieve a more three-dimensional characterization of the data; then, based on the dynamic metadata and knowledge graph of the target user, determine the data to be pushed of the target user, and push the data to be pushed to the target user. It can be seen that the data weaving method provided in the embodiment of the present application can actively discover the business needs of users, and recommend business-related data to users in combination with the knowledge graph, so that users can obtain the required data in real time and accurately, improving the user experience.

[0041] See also Figure 1 , is an architecture diagram of the data weaving system provided in the embodiment of the present application. Figure 1 As shown, the data weaving system 100 includes: a collection module 101 , a construction module 102 , an active metadata module 103 and an application module 104 .

[0042] The acquisition module 101 is used to acquire data from various data sources.

[0043] In some embodiments, the multiple data sources include, but are not limited to: data lakes, data warehouses, relational databases, cloud storage, and application systems.

[0044] In some embodiments, the raw data in the various data sources may include structured data, semi-structured data, or unstructured data. For example, structured data may be data from databases such as enterprise resource planning (ERP), common database (CDB), and cloud data warehouse (CDW). Semi-structured data may be data such as Json text and Log logs. Unstructured data may be data such as documents, pictures, and videos.

[0045] Among them, for unstructured data, the method provided in the embodiment of the present application can interpret the content of unstructured data based on machine learning algorithms, so as to identify metadata from the unstructured data, enrich the business semantics of the metadata, and prepare the content for building a knowledge graph by building module 102.

[0046] In some embodiments, the data collected from multiple data sources include metadata, crowdsourced data, and business context information. Exemplarily, metadata includes, but is not limited to, technical metadata, business metadata, operational metadata, social metadata, and localized knowledge (where localized knowledge refers to a knowledge graph that has been built and stored locally). Crowdsourced data includes, but is not limited to, user behavior, user comments, user ratings, and certifications. Business context information includes, but is not limited to, business terms, business policies, and data dictionaries.

[0047] In some embodiments, the data collected from various data sources include: static metadata and dynamic metadata. Static metadata refers to metadata related to the results of the data, such as user attribute information and other data. Dynamic metadata refers to metadata related to the state and use of the data and in a continuously changing state, such as data related to user behavior.

[0048] In some embodiments, Figure 2 As shown, the acquisition module 101 includes a collector, an acquisition configuration unit, a receiving processing unit, a storage adapter unit and an external interface.

[0049] The collector is used to collect data from the data source according to the information configured by the collection configuration unit.

[0050] For example, Figure 2 As shown, the types of collectors include: plug-ins, scripts, crawlers, etc. The collector is integrated in the data source, and the collector can transmit the collected data to the receiving processing unit through the interface. For example, the interface between the collector and the interface processing unit includes: message interface, representational state transfer (REST) ​​interface, application program interface (API), etc.

[0051] The collection configuration unit is used to interact with the user and supports the user to configure the source, type, model used and other information of the collected data.

[0052] The receiving and processing unit is used to receive the data collected by the collector and perform data analysis. Exemplarily, the data analysis includes analyzing data source information, metadata structure information, metadata association information, metadata specification checking, and metadata classification.

[0053] Optionally, for semi-structured data, the receiving and processing unit may first use a wrapper to convert the semi-structured data into structured data; and then perform the above data analysis on the structured data.

[0054] Optionally, for unstructured data, the receiving processing unit may not process it, and may call the storage adaptation unit to send the unstructured data to the construction module 102, which will extract, fuse, process and store it.

[0055] The storage adapter unit is used to store the analyzed and processed metadata into the graph database.

[0056] The construction module 102 is used to construct a knowledge graph based on the data collected by the collection module 101.

[0057] Exemplarily, the construction module 102 may construct a knowledge graph by combining top-down ontology design and bottom-up knowledge extraction.

[0058] In some embodiments, Figure 3 As shown, the process of constructing the knowledge graph by the construction module 102 includes: knowledge modeling, knowledge extraction, knowledge fusion, knowledge processing, etc.

[0059] Among them, knowledge modeling refers to building an ontology. For example, business experts and industry experts in the field can collaborate to develop and build a standardized domain ontology.

[0060] Among them, knowledge extraction refers to extracting entities, attributes and relationships from the ontology.

[0061] In some embodiments, entity parallel relationship similarity calculation and entity hypernym relationship extraction can be performed based on natural language processing (NLP) technology to abstract the ontology from the bottom up.

[0062] For example, knowledge extraction can be performed through an extraction model. First, business experts use corpus annotation tools to complete manual corpus annotation under the specifications of the domain ontology. At the same time, with the help of remote supervision and other technologies, semi-automatic corpus annotation is completed as auxiliary corpus annotation to provide data and labels for training extraction models. Secondly, based on the annotated corpus and machine learning algorithms, extraction model training for data cleaning, entity extraction, attribute extraction and relationship extraction is performed. Based on the trained model, entity extraction, attribute extraction and relationship extraction of metadata are automatically completed.

[0063] Knowledge fusion refers to the fusion of descriptive information about the same entity or concept from multiple data sources. For example, based on the extracted entities, attributes and relationships, with the support of the ontology library and rule library, entity matching, entity alignment, entity classification and ontology fusion can be completed to ensure the rigor and attribute consistency of the knowledge graph.

[0064] Knowledge processing can use knowledge reasoning methods to achieve knowledge completion and knowledge error correction, and business experts can evaluate and optimize the quality of the graph.

[0065] After knowledge modeling, knowledge extraction, knowledge fusion, and knowledge processing, the construction module 102 can construct a knowledge graph corresponding to the metadata based on the extracted entities, attributes, and relationships.

[0066] In some embodiments, Figure 3 As shown, the construction module 102 also includes a database, and the construction module 102 is also used to store the knowledge graph in the database after constructing the knowledge graph corresponding to the metadata.

[0067] Optionally, the database may include a graph database and a relational database; the knowledge graph may be stored in a graph database and / or a relational database. For example, the knowledge graph may be stored entirely in a graph database; or, the knowledge graph may be stored entirely in a relational database; or, the entities and relationships in the knowledge graph may be stored in a graph database, and the attributes in the knowledge graph may be stored in a relational database.

[0068] The graph database may include a graph engine function and a graph storage function. The graph storage function refers to the function of storing knowledge graphs; the graph engine function refers to the function of querying and analyzing the stored knowledge graphs.

[0069] The relational database may include a variety of data tables for storing data relationships. It is understandable that since the relational database is in the form of data tables, the relational database cannot store graph topology structures.

[0070] The active metadata module 103 is used to provide personalized metadata recommendation services to users in different business fields and data projects.

[0071] In some embodiments, Figure 4 As shown, the data recommendation process of the active metadata module 103 includes: data recall, data sorting, and data rearrangement.

[0072] In some embodiments, data recall recalls data that the user desires to recall from the raw data of the multiple data sources based on a recall model.

[0073] Among them, the recall models include: data collaborative filtering (Iterm-CF) model based on collaborative filtering (CF) algorithm, user collaborative filtering (User-CF) model, graph embedding model, multi-interest network with dynamic routing (MIND) model, crowd-based filtering model, etc.

[0074] In some embodiments, the data ranking is based on a ranking model for determining a ranking of the recall data.

[0075] Among them, the sorting models include: feature engineering model, logistic regression (LR) model, behavior sequence transformer (BST) model, deep factorization machine (Deep FM) model, deep residual network (DeepResNet) model, etc.

[0076] In some embodiments, the data shuffling adjusts the ranking of the recall data based on the shuffling model.

[0077] The reordering model is a model built based on manually set rules; or, the reordering model is a model built based on a machine learning strategy. It is understandable that the reordering model needs to be built and trained based on human experience and human understanding of the business.

[0078] Optionally, the data rearrangement mechanism may include at least one of the following: rearrange the data after deduplication and fragmentation, rearrange the data according to the diversity of the data, rearrange the data after multi-task fusion, rearrange the data after improving the real-time performance of the rearrangement model, and rearrange the data according to rule strategies, etc.

[0079] Exemplarily, data rearrangement includes: breaking up the recommended data into multiple dimensions as needed, improving the real-time performance of the rearrangement model (improving the real-time performance of the rearrangement model means: quantizing and trimming the rearrangement model to optimize the performance and efficiency of the rearrangement model), and obtaining the final data recommendation result. For example, a standard category system is established in combination with the knowledge graph for data category expansion at the semantic level.

[0080] It should be noted that data rearrangement is an optional step, which can be designed specifically according to business needs or incorporated into the data sorting step.

[0081] The application module 104 is used to recommend data to users of the third-party service platform according to the needs of target users of the third-party service platform.

[0082] In some embodiments, Figure 5 As shown, the data weaving system 100 provided in the embodiment of the present application can be integrated with a third-party business platform to obtain the needs of the target users of the third-party business platform, determine the target users' data to be recommended according to the needs of the target users of the third-party business platform, and recommend the data to be recommended to the target users.

[0083] As a possible implementation, a third-party business platform can be integrated into the data weaving system 100. In this way, a user can complete data services such as service sharing, business intelligence (BI) reports, and data quality management by operating a context in the data weaving system 100, and the data weaving system 100 can provide metadata search, recommendation, classification, lineage and other functions for the third-party business platform.

[0084] As another possible implementation, the data weaving system 100 can be integrated into a third-party business platform. In this way, the user can search for metadata and view personalized recommended metadata in the third-party business platform. At the same time, the user can jump to the corresponding metadata interface to view data and apply for data by clicking on the queried metadata or personalized recommended metadata.

[0085] Optionally, the data weaving system provided in the embodiment of the present application can be integrated with a third-party business platform through various technologies such as Java database connectivity (JDBC), REST, API interface, and messaging.

[0086] In some embodiments, the application module 104 includes: a service configuration unit, a service management unit, a rule management unit, and a recommendation implementation unit.

[0087] The service configuration unit is used to display the basic information configuration embedded in the third-party business platform to users in a visual form, including the application address, icon, login, authentication and other information of the third-party business platform. The service configuration unit also supports single sign-on and permission management for docking with the third-party business platform.

[0088] It is understandable that the third-party business platform integration solves the business island problem. The third-party business platform processing can not only provide metadata to the data weaving system 100, but also use the metadata and active recommendation functions of the data weaving system 100. In this way, users can find and use data in one context.

[0089] The service management unit is used to manage third-party applications that have been integrated into the system and set the access rights of users of the data weaving system to third-party business platforms. For example, through the service management unit, the administrator can view the third-party business platforms that have been integrated into the data weaving system and configure different access rights for different users.

[0090] The rule management unit is used to obtain the intelligent service selected by the third-party business platform and the configuration related to the intelligent service, and complete the initialization of the intelligent service environment. Exemplarily, the configuration related to the intelligent service includes but is not limited to: configuring the service type, service parameters, service implementation interface, third-party information collection configuration, etc. The rule management unit calculates and applies for resources based on the parameter information configured by the third-party business platform.

[0091] By way of example, the service types of the third-party business platform may include at least one of the following: a data analysis recommendation platform, a security and compliance management platform, an intelligent collection and orchestration platform, and an intelligent operation and maintenance recommendation platform.

[0092] Among them, the data analysis recommendation platform can actively recommend data to users based on their data needs, reducing the workload of users in data discovery. The security and compliance management platform can intelligently classify and grade data, and then protect the data according to the results of classification and grading. For example, manage user permissions based on the results of data classification and grading; or, encrypt data based on the results of data classification and grading; or, desensitize data based on the results of data classification and grading (for example, obfuscate, replace, empty, encrypt, etc. sensitive data). The intelligent collection and orchestration platform can be used to recommend data pipeline orchestration to users and automatically and dynamically allocate resources. The intelligent operation and maintenance recommendation platform can be used to proactively prevent network alarms and user complaints and identify equipment hidden dangers.

[0093] The recommendation implementation unit is used to obtain information from the third-party business platform in real time (for example, request information, model information, function information, access control information, etc.), and then trigger the active metadata module 103 through a trigger to provide personalized data recommendation services to users. Exemplary personalized data recommendation services include but are not limited to: data analysis recommendations, security and compliance management recommendations, intelligent collection and orchestration recommendations, and intelligent operation and maintenance recommendations.

[0094] Optionally, the recommended implementation unit serves as an adaptation layer of the data weaving system 100, which can shield differences in intelligent data service requirements of different third-party business platforms, including but not limited to: differences in input interface types, formats, and output result processing methods.

[0095] Optionally, the data weaving system 100 can provide metadata query services to third-party business platforms. Exemplarily, the third-party business system can query metadata through Kafka, REST, API, etc. The data query content includes but is not limited to: data, users, business classification, data quality, data statistics, etc.

[0096] It should be noted that the system architecture and application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0097] The following is a detailed introduction to a data weaving method provided in an embodiment of the present application.

[0098] For example, Figure 6 As shown, a data weaving method provided in an embodiment of the present application can be implemented as follows:

[0099] S201. Collect data from multiple data sources respectively.

[0100] The data collected from various data sources include: dynamic metadata and static metadata.

[0101] In some embodiments, the data collected from multiple data sources also include at least one of the following: metadata representing data source information, metadata representing business attributes, user behavior data, user preference data, user attribute data, and business context information.

[0102] The metadata that characterizes the data source information can also be called technical metadata. For example, technical metadata may include data source information, database table names and field names, field attributes, physical data models, processing process (such as extraction, cleaning, transformation, loading (ETL) operation process) details, data lineage documents (including information on the impact of upstream and downstream changes, data access permissions, groups, roles), etc.

[0103] Metadata that characterizes business attributes can also be called business metadata. Exemplarily, business metadata may include: business definition of data sets, explanation of business terms, business indicator names, calculation calibers, derived indicators, rules of business rule engines, data quality detection rules, data mining algorithms, data security levels or sensitivity levels, etc.

[0104] User behavior data may include: number of visits / applications, page browsing time, visit time, page dwell time, browsing path, etc.

[0105] User preference data may include: login method, browsing content, comment content, interactive content, favorite content, data preferences, etc.

[0106] User attribute data may also be referred to as user basic information data. For example, user basic information data may include: name, gender, date of birth, place of origin, job title, department, authority, active time, etc.

[0107] Business context information may include: business terms, business policies, and data dictionaries.

[0108] Optionally, the data collected from various data sources may also include operational metadata. Exemplarily, the operational metadata may include: data access method, access time, access restrictions, data access rights, groups and roles, results of data processing jobs, system execution logs, data backup, archiver, archive time, etc.

[0109] Optionally, the data collected from various data sources may also include content enhancement metadata. Exemplarily, the content enhancement metadata may include: electronic document data extracted by interpreting the content of unstructured data based on a machine learning algorithm. For example, the content enhancement metadata may be metadata such as keywords, tags, summaries, semantic relevance, and text similarity of a sales contract.

[0110] As a possible implementation method, the above step S201 can be implemented as follows: collecting technical metadata from database data sources such as mysql, oracle, elasticsearch, etc. from the business system through Kafka, REST, API, etc.; collecting business metadata, management metadata, social metadata, crowdsourcing metadata, etc. from BI tools, ETL tools, governance tools, applications (Application, APP), etc.

[0111] It is understandable that the method provided in the embodiment of the present application adopts an open metadata collection framework and sets up a large number of metadata collectors for the data ecosystem. In this way, new data sources can be quickly accessed at low cost to form data context information with complete links (for example, real-time, detailed, and accurate full-link lineage).

[0112] In some embodiments, the raw data in the various data sources include: structured data, semi-structured data, and unstructured data.

[0113] In some embodiments, the above-mentioned multiple data sources include a third-party business platform. The above-mentioned step S201 can be implemented as: collecting data of the target user from the third-party business platform. Wherein, the target user is a user of the third-party business platform. The data of the target user includes dynamic metadata of the target user.

[0114] S202. Build a knowledge graph based on data collected from multiple data sources.

[0115] In some embodiments, the above step S202 can be implemented as the following steps:

[0116] Step a1: Construct an ontology based on data collected from various data sources.

[0117] For example, business experts and industry experts in the field can collaborate to develop and construct a standardized domain ontology.

[0118] Step a2: Extract entities, attributes and relationships from the ontology.

[0119] In some embodiments, entity parallel relationship similarity calculation and entity hierarchical relationship extraction can be performed based on NLP technology to abstract the ontology from the bottom up.

[0120] Exemplarily, knowledge extraction can be performed through an extraction model. First, business experts use corpus annotation tools to complete manual corpus annotation under the specifications of the domain ontology. At the same time, with the help of remote supervision and other technologies, semi-automatic corpus annotation is completed as auxiliary corpus annotation to provide data and labels for training extraction models. Secondly, based on the annotated corpus and machine learning algorithms, extraction model training for data cleaning, entity extraction, attribute extraction and relationship extraction is performed. Exemplarily, the basic model for entity extraction can be a bidirectional Transformer encoder (bidirectional encoder representations from transformers, BERT), a bidirectional long short-term memory network (bi-directional long short-term memory, BiLSTM), and a conditional random field (conditional random fields, CRF). Based on the trained model, entity extraction, attribute extraction and relationship extraction of metadata are automatically completed.

[0121] In some embodiments, the above method may further include: based on the extracted entities, attributes and relationships, with the support of the ontology library and the rule library, completing entity matching, entity alignment, entity classification and ontology fusion. In this way, the rigor and attribute consistency of the knowledge graph can be ensured.

[0122] In some embodiments, the above method may also include: using knowledge reasoning methods to achieve knowledge completion and knowledge error correction, and business experts evaluating and optimizing the quality of the graph.

[0123] Step a3: Build a knowledge graph based on the extracted entities, attributes and relationships.

[0124] It is understandable that the knowledge graph construction method provided in the embodiment of the present application is based on static metadata and dynamic metadata collected from multiple data sources, constructs metadata knowledge graphs in a top-down and bottom-up manner, and quantifies entity connections and connection relationships based on artificial intelligence or machine learning algorithms. In this way, it is possible to automatically mine and establish associations between metadata to form a semantic knowledge graph, thereby achieving a more three-dimensional characterization of data and supporting more intelligent data usage recommendations.

[0125] To facilitate understanding, the knowledge graph construction process provided in the embodiment of the present application is described below in the form of an example.

[0126] For example, Figure 7 As shown in the figure, the data processing pipeline is used as an example to illustrate the construction process of the knowledge graph:

[0127] First, domain experts build ontology based on business analysis, such as data source, data set, processing process, user, user group, etc. Among them, the input and output of the processing process are data sets, the owner of the data set is the user, and the user is a member of the user group, etc.

[0128] Secondly, entities, relationships and attributes are extracted from the collected metadata based on the ontology and machine learning algorithms designed by domain experts. Figure 7 The entities, relationships and attributes of the processing process 1 shown in the figure include: the structured query language (SQL) corresponding to the processing process, the data source of the processing process, the input data set, the output data set and the blood relationship, etc.

[0129] Next, the extracted knowledge is integrated with the existing knowledge, for example Figure 7 The processing process 1 shown in the figure performs entity matching and user clustering with dataset 1 and dataset 2 to complete ontology fusion.

[0130] Finally, the metadata knowledge graph constructed based on the above entities, attributes and relationships is saved in the graph database after expert quality assessment and optimization.

[0131] S203: Determine the data to be pushed of the target user based on the dynamic metadata and knowledge graph of the target user.

[0132] Exemplarily, the dynamic metadata of the target user may be the access behavior metadata of the target user.

[0133] In some embodiments, Figure 8 As shown, the above step S203 can be implemented as the following steps:

[0134] S2031. Input the dynamic metadata and knowledge graph of the target user into M recall models to obtain at least one recall data.

[0135] The recall model is used to recall the data that the user expects to recall from the original data of multiple data sources; M is an integer greater than or equal to 1.

[0136] Exemplarily, the recall strategy based on the graph embedding model can be implemented as follows: based on the user access behavior metadata, the graph embedding model is used to generate the embedding vector of the data; and according to the embedding vector of the data, the similarity between the data is calculated; and then the recalled data is determined according to the similarity between the data. For example, it can be implemented as the following steps:

[0137] Step b1: construct a directed graph of data based on user access behavior metadata.

[0138] Step b2: Select a starting point from the directed graph, and then use a deep random walk algorithm to generate a data sequence.

[0139] Step b3: Introduce supplementary information (side information) of the data based on the knowledge graph.

[0140] Exemplarily, the supplementary information of the data includes: content enhancement metadata such as category, description, keywords, etc. to enrich the representation, and train the Word2vec model to generate embedding vector representation of the data.

[0141] Step b4: Calculate the similarity score between the two data based on the embedding vector representation and generate an i2i table.

[0142] Step b5: query the i2i table through user triggers to recall candidate recommendation data.

[0143] It should be noted that the embodiments of the present application provide a variety of recall strategies, and different recall strategies can be selected according to different scenarios. For example, a single-channel recall strategy of a recall model is selected. Alternatively, multiple recall models are selected for multi-channel recall, and a multi-channel recall strategy that fuses multi-channel recall data (it is understandable that the multi-channel recall strategy can improve the accuracy of the recall data). Alternatively, a recall strategy of a custom recall model is adopted.

[0144] Exemplarily, the method provided in the embodiment of the present application can group users with similar observation patterns together to recommend similar content. The similarity between different users is calculated based on the user's behavioral metadata, and then the user groups are clustered by preset thresholds or preset rules. According to the historical behavioral metadata of each type of user group, the data with a high access or application rate of the user group is recalled as the candidate recommendation data of the user group. By manually setting empirical parameters and rules, the data recalled based on the graph embedding algorithm and the data recalled based on user clustering are fused as the final candidate recommendation data.

[0145] S2032. Determine the data to be pushed to the target user according to at least one recalled data.

[0146] As a possible implementation manner, the above step S2032 may be implemented as: taking at least one recalled data as the data to be pushed to the target user.

[0147] As another possible implementation, the data collected from the multiple data sources includes the target user's data. The step S2032 can be implemented as the following steps:

[0148] Step c1: extracting the target user's features from the target user's data.

[0149] The characteristics of the target users include the target users' behavioral characteristics, attribute characteristics, and preference characteristics.

[0150] Step c2: Input the target user's features and at least one recalled data into N ranking models to obtain a ranking of at least one recalled data.

[0151] Among them, the sorting model is used to determine the matching degree between each recalled data and the characteristics of the target user, and determine the ranking of the recalled data according to the matching degree.

[0152] Step c3: determining the data to be pushed to the target user from the at least one recalled data according to the ranking of the at least one recalled data.

[0153] For example, taking the sorting model as the Deep FM model as an example, the sorting strategy based on the Deep FM model is described:

[0154] Step d1: extract user features based on the collected data set.

[0155] Exemplarily, user features include: user attribute features, user behavior features, user preference features, data-side features, interaction features, context features and other features. Among them, user attribute features may include user basic feature metadata, such as the user's level, position and department. User behavior features may include the data sequence accessed by the user, the data sequence queried by the user, the access time and the corresponding position deviation. User preference features may include data reflecting user preferences, such as user access time data, user search content data, and user click content data. Data-side features may include basic feature metadata of data, such as data identity (ID), access volume, source, creation time, and content enhancement metadata, such as keywords, categories, and themes of data. Other features such as interaction features and context features can be extracted according to the needs of the application field.

[0156] As a possible implementation, user features may be extracted based on a feature extraction model. Exemplarily, the collected data set is input into a trained feature extraction model to obtain user features.

[0157] It should be noted that the application embodiment provides a variety of feature engineering algorithms, including but not limited to feature extraction, dimensionality reduction and processing. In the embodiment of the present application, a suitable feature extraction algorithm can be selected according to different scenarios or a custom feature engineering model can be imported; or, a plurality of feature extraction algorithms can be combined to extract user features, and then feature fusion can be performed based on a fusion strategy to obtain the final user features.

[0158] Step d2: Train the Deep FM model based on user characteristics.

[0159] Exemplarily, user-side features, user behavior features, data-side features, interaction features, context features, and other features are processed and fused, and then input into the DeepFM model together with labels (0 indicates no visit, 1 indicates visit) for training.

[0160] Step d3: Based on the trained DeepFM model, predict the user's next click data.

[0161] Step d4: Determine the ranking of the recalled data based on the prediction results of the DeepFM model.

[0162] It should be noted that the framework of the present invention provides a variety of feature engineering algorithms, including but not limited to feature extraction, dimensionality reduction and processing. Appropriate feature extraction algorithms can be selected according to different scenarios or custom feature engineering models can be imported. Multiple feature extraction algorithms can be combined to perform feature fusion through artificially designed fusion strategies. Similarly, the framework of the present invention provides a variety of sorting models, and different sorting models can be selected according to different scenarios or custom sorting models can be imported.

[0163] It should be noted that the embodiments of the present application provide a variety of sorting models, and different sorting models can be selected according to different scenarios or custom sorting models can be imported.

[0164] In some embodiments, the above step c3 can be implemented as the following steps:

[0165] Step c3-1: Determine at least one candidate data from the at least one recalled data according to the ranking of the at least one recalled data.

[0166] The at least one candidate data is data in the at least one recalled data whose ranking is greater than a first preset threshold. Exemplarily, assuming that there are 500 at least one recalled data, the top 100 recalled data in the at least one recalled data can be used as at least one candidate data.

[0167] Step c3-2, perform data processing on at least one candidate data to obtain at least one processed candidate data; wherein the data processing includes at least one of the following: fragmentation, deduplication, fusion (such as multi-task fusion), diversity extraction, real-time improvement, etc.

[0168] Step c3-3: adjust the ranking of at least one candidate data according to the processed at least one candidate recall data.

[0169] Step c3-4, according to the adjusted ranking of at least one candidate data, determine the data to be pushed to the target user from the at least one candidate data. Exemplarily, the candidate data whose ranking in the at least one candidate data after adjustment is greater than the second preset threshold is determined as the data to be pushed to the target user. For example, assuming that there are one hundred at least one candidate data, the top ten candidate data in the at least one candidate data after adjustment can be used as the data to be pushed to the target user.

[0170] It is understandable that, since the data weaving method provided in the embodiment of the present application can be applied to different third-party business platforms, the embodiment of the present application supports selecting different data processing methods to process at least one recalled data according to different business needs; and then adjusting the ranking of at least one recalled data according to the processed at least one recalled data. In this way, the data to be pushed to the target user determined according to the ranking of at least one recalled data is more in line with the business needs of the target user, and the accuracy of the recommendation results is improved.

[0171] S204: Push the data to be pushed to the target user.

[0172] In some embodiments, the above step S204 may be implemented as: sending the data to be pushed to a third-party service platform, so that the third-party service platform pushes the data to be pushed to the target user.

[0173] Exemplarily, the data weaving system can send the data to be pushed to the third-party business platform through an interface. Exemplarily, the above interface can be JDBC, REST, API interface or message, etc.

[0174] It can be understood that the method provided in the embodiment of the present application can collect data from multiple data sources (wherein the multiple data sources include a third-party business platform, and the data collected from the multiple data sources include the data of the target user), and the collected data includes static metadata and dynamic metadata, which can enrich the business semantics of the data; further, a knowledge graph is constructed based on the data collected from multiple data sources, which can actively identify and establish data associations, and achieve a more three-dimensional characterization of the data; then, based on the dynamic metadata and knowledge graph of the target user, the data to be pushed of the target user is determined, and the data to be pushed is pushed to the target user. It can be seen that the data weaving method provided in the embodiment of the present application can actively discover the business needs of users, and recommend business-related data to users in combination with the knowledge graph, so that users can obtain the required data in real time and accurately, and improve the user experience.

[0175] In some embodiments, the third-party business platform includes at least one of the following: a data analysis recommendation platform, a security and compliance management platform, an intelligent collection and orchestration platform, and an intelligent operation and maintenance recommendation platform.

[0176] Exemplarily, in the case where the third-party business platform is a data analysis recommendation platform, taking smart asset recommendation as an example, the method provided in the embodiment of the present application can combine metadata collected from multiple data sources to characterize the relationship between users, data and businesses, and recommend the data sets that may be of most interest to each user, thereby greatly improving data discovery efficiency.

[0177] Exemplarily, in the case where the third-party business platform is a security and compliance management platform, taking intelligent data classification as an example, the method provided in the embodiment of the present application can be based on metadata and field content sampling collected from multiple data sources, to achieve automatic identification of sensitive information, complete accurate data classification and grading, and combine security policies to perform security and compliance management of data access, sharing and transmission processes.

[0178] Exemplarily, in the case where the third-party business platform is an intelligent collection and orchestration platform, taking intelligent operation and maintenance recommendation as an example, the method provided in the embodiment of the present application can actively identify equipment hidden dangers based on metadata collected from multiple data sources, predict user complaints and network traffic, etc., and then actively prevent network alarms and user complaints.

[0179] Exemplarily, in the case where the third-party business platform is an intelligent operation and maintenance recommendation platform, taking intelligent collection and orchestration as an example, the method provided in the embodiment of the present application can recommend data pipeline orchestration methods to users based on metadata collected from multiple data sources, automatically allocate resources, and quickly support current and future needs.

[0180] It is understandable that, in order to realize the above functions, the data weaving device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiments of the present application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0181] The embodiment of the present application can divide the functional modules of the data weaving device according to the above method embodiment. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one functional module. The above integrated module can be implemented in the form of hardware or software. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. The following is an example of dividing each functional module corresponding to each function.

[0182] Fig. 9 is a schematic diagram of the structure of a data weaving device provided in an embodiment of the present application. The data weaving device can execute the data weaving method provided in the above method embodiment. Fig. 9 As shown, the data weaving device 300 includes: a collection module 301 , a determination module 302 and a push module 303 .

[0183] The collection module 301 is used to collect data from multiple data sources respectively, and build a knowledge graph based on the data collected from the multiple data sources; wherein the data collected from the multiple data sources include dynamic metadata and static metadata.

[0184] The determination module 302 is used to determine the data to be pushed of the target user based on the dynamic metadata and knowledge graph of the target user.

[0185] The push module 303 is used to push the data to be pushed to the target user.

[0186] In some embodiments, the determination module 302 is specifically used to input the dynamic metadata and knowledge graph of the target user into M recall models to obtain at least one recall data; wherein the recall model is used to recall the data that the user expects to recall from the original data of multiple data sources; M is an integer greater than or equal to 1; and the data to be pushed of the target user is determined based on at least one recall data.

[0187] In some embodiments, the data collected from multiple data sources include data of the target user; a determination module 302 is specifically used to extract characteristics of the target user from the target user's data; wherein the characteristics of the target user include behavioral characteristics, attribute characteristics, and preference characteristics of the target user; the characteristics of the target user and at least one recalled data are input into N sorting models to obtain a ranking of at least one recalled data; the sorting model is used to determine the degree of match between each recalled data and the characteristics of the target user, and to determine the ranking of the recalled data based on the degree of match; based on the ranking of at least one recalled data, the data to be pushed of the target user is determined from the at least one recalled data.

[0188] In some embodiments, the determination module 302 is specifically used to determine at least one candidate data from at least one recalled data based on the ranking of at least one recalled data; the at least one candidate data is data whose ranking in at least one recalled data is greater than a preset threshold; data processing is performed on the at least one candidate data to obtain at least one processed candidate data; the data processing includes at least one of the following: breaking up, deduplication, and fusion; adjusting the ranking of at least one candidate data based on the processed at least one candidate data; determining the data to be pushed to the target user from the at least one candidate data based on the adjusted ranking of the at least one candidate data.

[0189] In some embodiments, the target user is a user of a third-party business platform; the collection module 301 is used to collect the data of the target user from the third-party business platform; the data of the target user includes the dynamic metadata of the target user; the push module 303 is used to send the data to be pushed to the third-party business platform, so that the third-party business platform pushes the data to be pushed to the target user.

[0190] In some embodiments, the third-party business platform includes at least one of the following: a data analysis recommendation platform, a security and compliance management platform, an intelligent collection and orchestration platform, and an intelligent operation and maintenance recommendation platform.

[0191] In some embodiments, the data collected from multiple data sources also include at least one of the following: metadata representing data source information, metadata representing business attributes, user behavior data, user preference data, user attribute data, and business context information.

[0192] In some embodiments, the raw data in the various data sources include: structured data, semi-structured data, and unstructured data.

[0193] In the case of implementing the functions of the above-mentioned integrated modules in the form of hardware, the embodiment of the present application provides another possible structure of the data weaving device involved in the above-mentioned embodiment. Fig.10 As shown, the data weaving device 400 includes: a processor 402 and a bus 404. Optionally, the data weaving device 400 may further include a memory 401; optionally, the data weaving device 400 may further include a communication interface 403.

[0194] Processor 402 may be a processor that implements or executes various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of the present application. The processor 402 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of the present application. The processor 402 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0195] The communication interface 403 is used to connect with other devices via a communication network, such as Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0196] The memory 401 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0197] As a possible implementation, the memory 401 can exist independently of the processor 402, and the memory 401 can be connected to the processor 402 via a bus 404 to store instructions or program codes. When the processor 402 calls and executes the instructions or program codes stored in the memory 401, the data weaving method provided in the embodiment of the present application can be implemented.

[0198] In another possible implementation, the memory 401 may also be integrated with the processor 402 .

[0199] The bus 404 may be an extended industry standard architecture (EISA) bus, etc. The bus 404 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0200] Some embodiments of the present application provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium), which stores computer program instructions. When the computer program instructions are executed on a computer, the computer executes the data weaving method described in any of the above embodiments.

[0201] Exemplarily, the above-mentioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks or magnetic tapes, etc.), optical disks (e.g., compact disks (CD), digital versatile disks (DVD), etc.), smart cards and flash memory devices (e.g., erasable programmable read-only memory (EPROM), cards, sticks or key drives, etc.). The various computer-readable storage media described in this application may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0202] An embodiment of the present application provides a computer program product comprising instructions. When the computer program product is run on a computer, the computer is enabled to execute the data weaving method described in any one of the above embodiments.

[0203] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data weaving method, characterized in that: The method comprises: Collecting data from multiple data sources respectively, and constructing a knowledge graph based on the data collected from the multiple data sources; wherein the data collected from the multiple data sources include dynamic metadata and static metadata; Determine the data to be pushed for the target user based on the dynamic metadata of the target user and the knowledge graph; Push the data to be pushed to the target user.

2. The method according to claim 1, characterized in that Determining the data to be pushed of the target user based on the dynamic metadata of the target user and the knowledge graph includes: Input the dynamic metadata of the target user and the knowledge graph into M recall models to obtain at least one recall data; wherein the recall model is used to recall the data that the user expects to recall from the original data of the multiple data sources; and M is an integer greater than or equal to 1; The data to be pushed to the target user is determined according to the at least one recalled data.

3. The method according to claim 2, characterized in that The data collected from the multiple data sources include the data of the target user; and determining the data to be pushed of the target user according to the at least one recalled data includes: Extracting the characteristics of the target user from the data of the target user; wherein the characteristics of the target user include the behavior characteristics, attribute characteristics, and preference characteristics of the target user; Inputting the target user's features and the at least one recalled data into N ranking models to obtain a ranking of the at least one recalled data; the ranking model is used to determine the degree of matching between each of the recalled data and the target user's features, and determine the ranking of the recalled data according to the matching degree; According to the ranking of the at least one recalled data, the data to be pushed to the target user is determined from the at least one recalled data.

4. The method according to claim 3, characterized in that The step of determining the to-be-pushed data of the target user from the at least one recalled data according to the ranking of the at least one recalled data comprises: Determining at least one candidate data from the at least one recalled data according to the ranking of the at least one recalled data; the at least one candidate data is data whose ranking in the at least one recalled data is greater than a preset threshold; Performing data processing on the at least one candidate data to obtain the at least one processed candidate data; the data processing includes at least one of the following: breaking up, deduplication, and fusion; Adjusting the ranking of the at least one candidate data according to the processed at least one candidate data; According to the adjusted ranking of the at least one candidate data, the data to be pushed to the target user is determined from the at least one candidate data.

5. The method according to claim 1, characterized in that The target users are users of a third-party business platform; The data are collected from a variety of data sources, including: Collecting the target user's data from the third-party service platform; the target user's data includes the target user's dynamic metadata; The step of pushing the data to be pushed to the target user includes: The data to be pushed is sent to the third-party service platform, so that the third-party service platform pushes the data to be pushed to the target user.

6. The method according to claim 5, characterized in that The third-party business platform includes at least one of the following: a data analysis recommendation platform, a security and compliance management platform, an intelligent collection and orchestration platform, and an intelligent operation and maintenance recommendation platform.

7. The method according to claim 1, characterized in that The data collected from multiple data sources also includes at least one of the following: Metadata representing data source information, metadata representing business attributes, user behavior data, user preference data, user attribute data, and business context information.

8. The method according to claim 1, characterized in that The original data in the multiple data sources include: structured data, semi-structured data and unstructured data.

9. A communication device, characterized in that: include: a processor and a memory for storing instructions executable by the processor; The processor is configured to execute the instructions so that the communication device performs the data weaving method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the data weaving method according to any one of claims 1 to 8.

Images