Full image data processing method and device, equipment and storage medium

Abstract

The application provides a full-mapping data processing method and device, equipment and a storage medium, relates to the technical field of data map, and the method comprises the following steps: obtaining a target field model by uniformly performing graph modeling processing on initial data resources of one or more external physical engines; the target field model is characterized by graph resources and graph relationships; data service processing is performed based on the graph resources and graph relationships in the target field model; and the initial data resources and / or the service processing results of the data service processing are displayed based on OpenAPI. The application reduces the difficulty of metadata modeling and management, unifies the metadata management between different engines, and saves the overhead of synchronizing metadata information between different engines for the same piece of metadata in the data map system.

Description

Technical Field

[0001] This application relates to the field of data mapping technology, and in particular to a method, apparatus, device and storage medium for full-map data processing. Background Technology

[0002] Data mapping systems are typically used to manage complex and heterogeneous data within an enterprise. Related technologies usually model business data within relational engines; however, the business data in data mapping systems has inherent correlations, and existing solutions cannot adequately adapt to this modeling approach, resulting in complex modeling. Furthermore, existing solutions rely on multiple systems to meet different business needs, which incurs significant data synchronization overhead. Summary of the Invention

[0003] The purpose of this application is to provide a full-map data processing method, apparatus, device and storage medium, which reduces the difficulty of metadata modeling, unifies metadata management between different engines, and saves the overhead of synchronizing metadata information between different engines for the same metadata in a data map system.

[0004] In a first aspect, the present invention provides a method for processing full-map data, the method comprising:

[0005] A target domain model is obtained by performing unified graph modeling processing on the initial data resources of one or more external physics engines; the target domain model is represented by graph resources and graph relationships.

[0006] Data business processing is performed based on the graph resources and graph relationships in the target domain model;

[0007] Based on OpenAPI, the initial data resources and / or the business processing results of data business processing are displayed.

[0008] Secondly, the present invention provides a full-map data processing apparatus, the apparatus comprising:

[0009] The modeling processing module is used to perform unified graph modeling processing on the initial data resources of one or more external physics engines to obtain a target domain model; the target domain model is represented by graph resources and graph relationships.

[0010] The data service processing module is used to perform data service processing based on the graph resources and graph relationships in the target domain model;

[0011] The data display module is used to display the initial data resources and / or the business processing results of data business processing based on OpenAPI.

[0012] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the full-map data processing method of any of the foregoing embodiments.

[0013] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the full-map data processing method of any of the foregoing embodiments.

[0014] The full-map data processing method, apparatus, device, and storage medium provided in this application involve performing unified graph modeling on initial data resources collected from one or more external physical engines to obtain a target domain model. The target domain model is represented by graph resources and graph relationships. Data business processing is performed based on the graph resources and graph relationships in the target domain model. The initial data resources and / or the business processing results are then displayed using OpenAPI. This method, by performing unified graph modeling on initial data resources collected from one or more external physical engines, can generate a graph model (i.e., a target domain model represented by graph resources and graph relationships) from initial data resources from different external physical engines, thereby reducing the difficulty of data model (i.e., metadata) modeling and management. Furthermore, by performing data business processing on graph resources and graph relationships obtained through unified graph modeling, metadata management between different engines is unified, while saving the overhead of synchronizing metadata information between different engines for the same metadata in a data map system. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating a full-map data processing method provided in this application embodiment;

[0017] Figure 2 A structural diagram of a data modeling module provided in an embodiment of this application;

[0018] Figure 3 A structural diagram of a data map calculation module provided in an embodiment of this application;

[0019] Figure 4 A schematic diagram of a graph engine provided in an embodiment of this application;

[0020] Figure 5 A structural diagram of a full-map data processing device provided in an embodiment of this application;

[0021] Figure 6 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0024] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0025] Data mapping systems, also known as metadata systems, are typically used to manage the complex and heterogeneous data within an enterprise. They are currently used to implement important functions such as enterprise data management, data development, and data governance. Common data mapping systems include the open-source Apache Atlas, as well as enterprise-built data mapping platforms based on data warehouses.

[0026] There are two main types of existing technical solutions: open-source solutions such as Apache Atlas, and enterprise-built data maps. The latter are typically customized by enterprises based on their specific data asset management needs. Existing data map systems are generally designed based on traditional relational engines. For example, for the storage component, the metadata, tag information, and classification information of the dataset are typically abstracted into information tables in a relational database, and records from different information tables are linked through foreign keys, mapping tables, etc. For the service component, relational data engine query interfaces (such as JDBC) are typically used to query, aggregate, encapsulate, and expose service interfaces.

[0027] This type of data map, developed based on a traditional relational data engine, has several obvious shortcomings:

[0028] 1. Metadata modeling is complex: Metadata modeling involves many entity tables and mapping tables, such as the base dataset table (Dataset) used to store dataset metadata, the tag table (Tag) used to store tag information, and the intermediate table used to store the mapping relationship between the base dataset table and the tag table.

[0029] 2. Data redundancy: A typical data mapping system may simultaneously exist a MySQL database for business data storage and an Elasticsearch database for fast user data retrieval. Data must be stored in at least two copies, and data consistency between different storage is also a very complex problem.

[0030] 3. Complex Server-Side Logic: To meet various business needs for metadata management, complex processing logic needs to be encapsulated using database query interfaces (such as JDBC). A typical scenario is the creation of a new dataset (e.g., a Hive table), which involves creating corresponding metadata records in the MySQL base dataset table, creating the actual table in the corresponding physical engine (e.g., Hive), and creating records for searching in Elasticsearch. The dataset may also need to be bound to existing tags or categories, or new tags or categories may be created and bound to the current table, which corresponds to creating relevant associated records in the corresponding mapping tables. As can be seen, creating a new dataset involves multiple steps and relies on multiple components, making it prone to exceptions. A typical scenario is that the metadata records in the business database have been created, but the records for searching in Elasticsearch fail to be created, resulting in data inconsistency.

[0031] 4. Complex Data Lineage Implementation: Most existing data mapping systems integrate data lineage functionality. This function analyzes existing data jobs (such as SQL jobs in a data warehouse), extracts the upstream and downstream flow relationships of data, and abstracts them into a directed acyclic graph (DAG). This graph is then displayed to users through the platform. Common displays include upstream tables (lineage backtracking) and downstream tables (impact analysis) for a given data table. Additionally, some systems support data governance based on data lineage. For example, they can analyze the dependencies of a useless table to identify other useless tables in its chain, or perform impact analysis on a specific table to provide timely warnings when data quality issues arise. Systems designed with relational engines often fall short when performing this type of data lineage analysis. The system needs to periodically or in real-time collect data tasks and store the task information (e.g., SQL) in a business database. Then, during the lineage analysis phase, the task information is read from the business database, upstream and downstream dependencies are parsed, and the data is stored back into the business database or a graph database. This approach is complex, requiring data to be copied and processed across multiple systems. Lineage analysis is difficult to perform using a business database (usually relational), while using a graph database introduces data redundancy and consistency issues. Many data mapping systems, while integrating graph databases for lineage analysis, often only use them to solve lineage problems, further increasing system complexity while addressing the issue.

[0032] To address one or more of the aforementioned problems, embodiments of this application provide a full-map data processing method, apparatus, device, and storage medium, which reduces the difficulty of metadata modeling and management, unifies metadata management between different engines, and saves the overhead of synchronizing metadata information between different engines for the same metadata in a data map system.

[0033] For ease of understanding, the full-map data processing method provided in the embodiments of this application will be described first, see [link to relevant documentation]. Figure 1 As shown, this full-map data processing method mainly includes the following steps:

[0034] Step S102: By performing unified graph modeling processing on the initial data resources of one or more external physics engines, a target domain model is obtained.

[0035] External physical engines can include those from the Hadoop ecosystem such as HDFS, HBase, Hive, Impala, Spark, Flink, MapReduce, Kudu, Elasticsearch, Kafka, and S3. Initial data resources can include at least one or more of the following: physical resources, data sources, data models, data tags, data processing tasks, business resources, and management resources.

[0036] The target domain model is represented by graph resources and graph relationships. The target domain model can be a graph model of the same format constructed from various data resources of different sources and types. Business processing of various data resources is performed through the graph resources and graph relationships corresponding to this graph model.

[0037] The aforementioned unified graph modeling process involves modeling different data from different external physical engines using a unified format, thereby enabling the resulting target domain model to represent the mapping relationships between different types of data in a consistent manner. In one implementation, metadata (also known as a data model), the relationships between metadata and tags, data processing flows, the relationship between data sources and data models, the relationship between data sources and physical resources, and the relationship between business processes, applications, and metadata can be abstracted into a graph model. This facilitates the processing of data from different external physical engines using a unified format.

[0038] Step S104: Perform data business processing based on graph resources and graph relationships in the target domain model.

[0039] The aforementioned graph resources typically include physical resources, data sources, data models, data labels, and data processing tasks. Depending on the requirements, unified graph modeling can also abstract graph resources at the business, application, and work order levels. For example, when a standard application uses big data resources, it reports application-level information through a data map, such as the types of businesses supported by the application and the corresponding work orders. After determining the graph resources, subsequent business processing logic can be based on these graph resources.

[0040] This embodiment abstracts mainstream databases and big data engines into graph resources, such as HDFS, HBase, Hive, Impala, Spark, Flink, MapReduce, Kudu, Elasticsearch, Kafka, and S3 within the Hadoop ecosystem. It also abstracts graph resources at the business, application, and ticket levels. For example, when a standard application uses big data resources, it reports application-level information through a data map, such as the business the application supports and the corresponding ticket information. It should be noted that due to the scalability of this invention's architecture, this invention is not limited to the abstracted graph resources described above; new graph resources can be easily incorporated into the system based on different requirements.

[0041] In practical applications, the aforementioned graph resources can be mapped to "points" in a data map graph engine. Within the data map graph engine, points can be defined using the interfaces provided by the graph engine, based on the definition of the graph resources. Each type of graph resource corresponds to a type of point in the graph engine. When using an attribute graph model, points can be attached to a series of attributes. Attributes can be key-value pairs, where the key represents the key and the value represents the value, with the value having a data type (such as int, float, string, etc.). For example, a Hive data source corresponds to a type of point in the graph, and this type of point may contain attributes such as the Hive engine version.

[0042] The aforementioned graph relationships include relationships between various graph resources. For example, there is a deployment relationship between physical resources and data sources; specifically, a Hive data source may be deployed on multiple physical resources. Another example is the inclusion relationship between data sources and data models; specifically, a Hive data source may contain multiple data models. Yet another example is the inclusion relationship between data models; for instance, a Hive data model may contain multiple data fields, and so on.

[0043] Generally speaking, the richer the graph relationships are defined, the stronger the graph engine's ability to handle complex business logic will be. For example, defining the relationship between business and physical engines can quickly determine the business that may be affected when the physical engine fails (such as when a machine crashes).

[0044] At the same time, defining more relationships is not necessarily better, because relationships correspond to edges in graph storage engines, and too many edges will cause the graph size to expand rapidly, leading to typical problems such as 'supernodes'. Supernodes are a typical problem in graph computing, which refers to the problem that when the number of edges adjacent to a node reaches a certain order of magnitude (such as hundreds of thousands or millions), the graph traversal performance through that node drops rapidly.

[0045] Graph relationships correspond to edge definitions in a data map graph engine. In the graph engine (usually a graph storage engine), based on the definitions of graph relationships, we define the corresponding edges using the interfaces provided by the graph engine. Each graph relationship corresponds to a type of edge in the graph engine. When using an attribute graph model, edges can also be attached to a series of attributes. The definition and use of attributes are similar to the attributes of the points mentioned above. For example, there might be an input / output relationship between a data model and a data processing task. This type of relationship corresponds to a type of edge in the graph, and the attribute included in this type of edge might be the last execution time (last_execute_time).

[0046] In one implementation, the aforementioned data service processing may include at least data lineage calculation. By performing complex data service processing such as data lineage calculation on the graph-based data, the processing of complex services can be simplified, and the processing efficiency of different data corresponding to different physical engines can be improved.

[0047] Step S106: Based on the OpenAPI, visualize the initial data resources and / or the business processing results of data business processing.

[0048] In one implementation, data visualization can be achieved through a client-side component consisting of two parts: a web interface and an OpenAPI. The web interface displays the data model (i.e., metadata) and / or the results of data processing, while the OpenAPI connects the data map's specified API interfaces to external systems via public APIs. In practice, the web interface can be implemented using mainstream web front-end frameworks, and the OpenAPI exposes certain APIs of the data map (which can be selected based on actual needs) to external systems as public APIs.

[0049] The full-graph data processing method provided in this application embodiment performs unified graph modeling processing on the initial data resources collected from one or more external physical engines. This allows the initial data resources from different external physical engines to be processed in a unified manner to generate a graph model (i.e., a target domain model representing graph resources and graph relationships), thereby reducing the difficulty of data model (i.e., metadata) modeling and management. By processing the graph resources and graph relationships obtained after unified graph modeling, the metadata management between different engines is unified, while saving the overhead of synchronizing metadata information between different engines for the same metadata in the data map system.

[0050] The above-mentioned method involves performing unified graph modeling on the initial data resources collected from one or more external physics engines to obtain a target domain model. In practical implementation, this may include the following steps:

[0051] Step 1.1) Obtain initial data resources from one or more external physical engines; wherein, the initial data resources include at least one or more of physical resources, data sources, data models, data tags, data processing tasks, business resources and management resources.

[0052] Step 1.2) involves transforming the initial data resources to obtain graph resources and graph relationships. In one implementation, the initial data resources can be cleaned first, and then data entities can be extracted and entity relationships mapped from the cleaned data to obtain graph resources and graph relationships. Graph resources are used to represent the data entities in the initial resources, and graph relationships are used to represent the relationships between data entities in a graphical form.

[0053] Step 1.3) Determine the target domain model through graph resources and graph relationships.

[0054] For ease of understanding, see [link to relevant documentation]. Figure 2 As shown, unified graph modeling can be performed using a collector, a mapper, and a pusher. The collector gathers initial data resources from an external physical engine; the mapper extracts entities and relationships from the initial data resources, maps them using a pre-established unified graph modeling model to obtain graph resources and graph relationships, and determines the target domain model based on these resources and relationships; the pusher pushes the target domain model to the data map graph engine.

[0055] The metadata collected by the metadata collector is cleaned and transformed before being stored in the data map storage module. The data map storage module can be implemented based on a graph database, specifically, the Dgraph graph database, which can support horizontal scaling. It has excellent performance for general queries under the scale of metadata and uses GraphQL as the query language, which fits the full graph architecture of this application embodiment. At the same time, it has good development in the open source community and facilitates subsequent docking and interaction with various external systems.

[0056] It should be noted that the database selection for the data map storage module used in this application is not limited to Dgraph; it can utilize currently mainstream attribute graph databases such as JanusGraph, using Gremlin as the primary query language (e.g., Neo4j), or Cypher as the query language, etc. In practical applications, these query languages ​​can be converted into GraphQL through various methods for subsequent storage and computation.

[0057] In one implementation, the graph database can be accessed through the RPC toolkit provided by the storage module to perform data CRUD operations and complex queries. For example, Dgraph provides SDKs in languages ​​such as Java and Python. By integrating the SDK into the metadata collector, the collector can easily send data to the graph database. It can also be integrated into subsequent service modules to implement complex business logic.

[0058] The aforementioned collectors include at least one or more of the following: HDFS collector, HBase collector, Hive collector, Impala collector, Spark collector, Flink collector, MapReduce collector, Kudu collector, Elasticsearch collector, Kafka collector, and S3 collector. These collectors are used to gather raw metadata from external physical engines.

[0059] Furthermore, this embodiment defines the development interface specifications for the data acquisition machine. By integrating a data acquisition machine with mainstream data engines, users can customize development to add a new engine based on the defined development interface specifications. The interface specifications are linked to graph resources and graph relationships. By defining a series of interface constraints, the same graph processing for different formats is achieved. This means converting data resources from different external physical engines into graph resources and graph relationships of a unified format through format conversion. For example, the interface specifications for the pusher to send data to the graph engine (input parameters, data format, output parameters, etc.), the mapping of newly added external engines to a new type of point in the graph engine and association with existing graph resources in the graph engine, and the data model specifying certain reserved attributes (such as version (new points in the graph engine will have a version attribute to track the evolution of the graph model)).

[0060] In an optional implementation, the aforementioned collector can be understood as an interface class that defines a series of attributes and methods. Attributes may include a unique ID, a unique name, and other key-value pairs of the collector. Methods may include: 1) `Init()` to initialize the collector; 2) `Collect()` to perform a single data collection; 3) `Push()` to push the collected data to the unified graph storage; 4) `Metrics()` to return the collector's status to the user; 5) `Start()` to start the collector; 6) `Suspend()` to temporarily pause the collector; and 7) `Stop()` to stop the collector. It is important to note that the `collect()` and `push()` methods typically require packaging the collected data into a specified format before pushing it. For example, in this invention, the collection time, data source, and metadata details need to be filled in. The `Push()` method accesses the unified graph storage using a RESTful approach or directly uses the graph storage SDK; therefore, the metadata collector also needs to configure the unified graph storage URL.

[0061] By using a pre-established initial unified graph modeling model, the initial data resources collected by the external physics engine are synchronized to the full-map data map system.

[0062] The aforementioned target domain model includes at least one or more of the following models: a first graph model obtained by extracting entities and relationships from a data model; a second graph model obtained by extracting entities and relationships from a data model and data labels; a third graph model obtained by extracting entities and relationships from data processing tasks; a fourth graph model obtained by extracting entities and relationships from a data source and a data model; a fifth graph model obtained by extracting entities and relationships from a data source and physical resources; and a sixth graph model obtained by extracting entities and relationships from business resources, data models, and management resources.

[0063] In an alternative implementation, the first graph model described above can be a graph model obtained by abstracting the data model (also known as metadata). For example, for tabular data (such as Hive tables), common metadata includes table name, table comments, field names, field types, field comments, etc. For instance, a mapping relationship can be constructed between table names and table fields, and between fields and field comments.

[0064] In an alternative implementation, the second graph model described above can be a graph model obtained by abstracting metadata and data tags. For example, in a financial system, multiple tables may have a tag called "payment details." By abstracting the tables and the "payment details" tag, the mapping relationship between each table and the tag can be obtained.

[0065] In an optional implementation, the third graph model described above can be a graph model obtained by abstracting the data processing flow. For example, offline tasks in a data warehouse are typically a series of SQL statements. By analyzing these SQL statements, input and output tables can be extracted, and the input tables, tasks, and output tables can be abstracted into a graph.

[0066] In an optional implementation, the fourth graph model described above can be a graph model obtained by abstracting the relationship between the data source and the data model. For example, the SQL statements in the third graph model are generally executed in the data warehouse. Specifically, the data warehouse engine may be a Hive data source. In this case, a connection can be established between the data source (Hive) and the data model, so that the fourth graph model can be used to determine which data source corresponds to which data model.

[0067] In an optional implementation, the fifth graph model described above can abstract the relationship between data sources and physical resources into a graph. For example, Hive data sources typically run as data engines on a group of physical machines (usually forming a cluster). These physical machines cooperate to form a cluster. We can establish a connection between the cluster resources corresponding to such physical machines and the data source, thereby determining which physical machine the corresponding data source is on through the fifth graph model.

[0068] In an optional implementation, the sixth graph model described above can be a graph model obtained by abstracting business resources, data models, and management resources. For example, if a business involves three data tables, the relationships between these three tables and the business and application can be established, so that when a business fails, the cause of the failure can be quickly located based on the abstracted graph relationships.

[0069] Furthermore, during data processing, the data map computation module can be used for corresponding processing. This module is a separate service from the aforementioned data map storage module, typically deployed in a different cluster. The data map computation module is generally stateless (although the computation tasks themselves may have state). Upon receiving an external request, the data map computation module retrieves the corresponding data from the data map storage module as needed, performs the computation within its own cluster, and returns the results asynchronously to the caller.

[0070] The graph computing framework for the full-map data map in this embodiment integrates the mainstream Pregel model and a series of toolkits for distributed computation of various complex business logics, such as offline computation of kinship relationships. Furthermore, the graph computing framework can also include distributed computing frameworks such as GraphLab and PowerGraph. Taking the Pregel model as an example, this model implements the BSP (Bulk Synchronous Parallel) computing model, and typical Pregel implementations include Apache Giraph.

[0071] Furthermore, the aforementioned data map computing module includes interconnected computing agent components and computing clusters, see [link to relevant documentation]. Figure 3 As shown. The computing agent component receives graph resources and graph relationships, and performs data service processing based on these resources and relationships. This data service processing includes at least one or more of the following: receiving, confirming, scheduling, distributing, and tracking the execution status of computing tasks. The computing cluster includes multiple computing nodes, which are used for distributed computing of graph resources and graph relationships through these nodes.

[0072] The computing agent component serves as the sole entry point for the computing service, acting as a medium for interaction with external programs. The agent is also responsible for receiving, acknowledging, scheduling, distributing, and tracking the execution status of computing tasks. These computing tasks can be short-term or long-term.

[0073] Short-duration computation tasks are typically computationally intensive tasks requested by the data map service module, usually taking only a few seconds to complete. Examples include querying the relationship between two graph resources (shortest path query). These queries are generally distributed to the computation module, which uses a dedicated computing cluster for computation. After receiving the task from the service module, the agent program determines the task type, selects the appropriate utility from its toolkit, and distributes the task parameters and the utility package together to the computing cluster. During computation, the computation module polls the status of the computation task. Once the computation task is complete, it retrieves the result and returns it to the calling program.

[0074] Long-duration computational tasks are typically routine background tasks of a data mapping system, or computationally intensive tasks requested by service modules, generally taking minutes or hours to complete. For example, incrementally calculating the bloodline of the entire map data daily requires retrieving the map data from the storage module, performing offline computation on the computing cluster, and then writing the final results back to the storage module. Another example is a data leaderboard feature that calculates a comprehensive metric based on the frequency of a dataset's use throughout the data chain to assess its popularity. This type of calculation involves calculating the in-degree and out-degree of points on the graph, often taking minutes, and is typically triggered asynchronously by service modules in the background.

[0075] In one specific implementation, the graph engine used in this embodiment for full-graph data processing can be found in [reference needed]. Figure 4 As shown, various external data collectors are used to collect metadata from external physical engines, including HDFS collectors, Hive collectors, HBase collectors, Kafka collectors, ES collectors, Flink collectors, Spark collectors, etc. After cleaning, transforming, and storing the data, the collectors input the data into the data map storage module and the data map calculation module for unified graph storage and unified graph calculation.

[0076] This embodiment separates the data map engine into two modules: a data map storage module and a data map computation module. This allows the data map storage module to focus on lightweight CRUD (Create, Read, Update, Delete) requests, while the computational tasks are scheduled to the computation module for asynchronous computation. Furthermore, it significantly separates OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing) jobs, preventing computational tasks requested by the service module from slowing down the entire storage cluster and causing some basic functions of the service module (such as basic metadata queries) to become unavailable.

[0077] In an optional implementation, when visualizing data, the server and client can interact via a web API, which uses the graph query language GraphQL. Since GraphQL is inherently graph-based, data is defined as an object type (also simply called a type) in the GraphQL model definition. Each type can have a series of fields, and a series of functions are defined to retrieve a specific field of a processed type. Types are generally customizable, but there are two special types: query and mutation. A query, defined within the GraphQL framework, is used to perform CRUD operations on data. GraphQL is integrated into the server and exposed externally via an HTTP interface. GraphQL has implementations in various programming languages, and when interaction with external systems is required, the GraphQL interface can be exposed to external callers.

[0078] In an alternative implementation, the server can use the Spring framework as the underlying foundation and integrate frameworks such as spring-graphql, graphql-java, and dgraph4j (the Java client for Dgraph) to implement business logic.

[0079] The aforementioned web interface also supports the following functions: 1. Displaying existing graph resources in form, for example, for Hive data tables, supporting the display of table structure (fields, field types, field comments), table description, table tags, etc.; 2. Supporting the creation of new graph resources, for example, users can create new Hive tables, requiring the specification of table name, field list, etc.; 3. Supporting the updating of existing graph resources, for example, for Hive data tables, supporting the modification of field types, field comments, etc.; 4. Supporting the classification management of graph resources. With the help of full graph modeling, graph resources can be bound to more abstract types. For example, Hive datasets can be categorized into a certain business domain. Currently, the system supports classification management based on subject domains; 6. Supporting the relationship management and analysis of graph resources. With the help of full graph modeling, various relationships can be established between graph resources, such as the inclusion relationship between data sources and data models disclosed in the aforementioned embodiments. This relationship has been modeled in the system backend and can therefore be easily obtained on the client side. In this way, the web interface can display, for example, which Hive databases are under a certain Hive data source, and which Hive data tables are under a Hive database. For example, the data ETL process can also be incorporated into a full graph system for management. The simplest data processing flow is as follows: an upstream data table, a processing task (such as a SQL statement), and a downstream data table. This is a typical graph relationship, and therefore can be modeled in a full graph system. The client can then obtain this graph relationship from the server. A typical scenario is the display of data lineage.

[0080] The OpenAPI supports the following functions: 1. Allowing users to define, create, and manage graph resources through the API; 2. Allowing users to manage the classification of graph resources through the API; 3. Allowing users to manage the tags of graph resources through the API; 4. Allowing users to manage and analyze the relationships of graph resources through the API.

[0081] In addition, to improve the efficiency of data service processing, in one implementation, graph resources can be classified to obtain the classification category to which the graph resources belong; wherein, the graph resources correspond to at least one category; in response to the input of the classification category, the graph resources in the target domain model are searched, and the corresponding graph relationships are matched according to the searched graph resources.

[0082] To facilitate providing users with a scalable data resource processing mechanism in subsequent processing, in an optional implementation, new data resources can be created in response to the creation operation of the graph engine; the new data resources can be classified and processed, and matched to the corresponding graph relationships in the target domain model based on the classification attributes.

[0083] In one implementation, when the external physical engine is the Hive engine, the corresponding graph resources include Hive data tables. In response to the creation operation of the graph engine, new data resources are created. In specific implementations, new graph resources can be created in response to the creation operation of table information (such as table name, table fields, list information, field types, and field comments) in the Hive data table.

[0084] Various relationships can be established between graph resources, such as the inclusion relationship between data sources and data models. This relationship is already modeled in the system backend and can therefore be easily accessed on the client side. In this way, based on the web, it is possible to display, for example, which Hive databases are under a certain Hive data source, and which Hive data tables are under a certain Hive database.

[0085] For example, the data ETL process can also be incorporated into a fully graph-based system for management. The simplest data processing flow is as follows: an upstream data table, a processing task (such as a single SQL statement), and a downstream data table. By performing unified graph modeling, the client can obtain this graph relationship from the server, and thus display the data lineage through the modeled graph.

[0086] Furthermore, the system architecture provided in this embodiment supports both pull and push metadata management modes. Pull mode refers to the data map platform actively pulling metadata from the corresponding physical engine using a metadata collector, performing ETL transformation, and then incorporating it into its own repository. Push mode allows users to push metadata to the data map system in a specified format via OpenAPI, and then display it through the platform. Pull mode is generally suitable for well-established storage engines, such as storage components in the Hadoop ecosystem, while push mode is more suitable for user-defined metadata, such as metadata from self-developed storage engines or applications.

[0087] In summary, the full-map data map system provided in this embodiment, implemented using a more full-map engine, helps enterprises build a one-stop graph-based data asset management platform. The system's underlying layer is based on graph engine technology, abstracting and integrating rich information such as domain models, business data, and system resources into a single system. This reduces the difficulty of data model building and the overhead of synchronizing metadata information across different engines for the same metadata in the data map system. The server-side uses a unified graph processing engine and exposes a unified graph interface, further reducing system complexity. Furthermore, the introduction of full-map functionality makes implementing complex business requirements such as data lineage much easier, significantly reducing development difficulty.

[0088] In response to the above-mentioned full-map data processing method, this embodiment provides a full-map data processing device, see [link to device]. Figure 5 As shown, the device mainly includes the following parts:

[0089] The modeling processing module 52 is used to perform unified graph modeling processing on the initial data resources of one or more external physical engines to obtain a target domain model; the target domain model is represented by graph resources and graph relationships.

[0090] Data business processing module 54 is used to perform data business processing based on graph resources and graph relationships in the target domain model;

[0091] The data display module 56 is used to display the initial data resources and / or the business processing results of data business processing based on OpenAPI.

[0092] The full-map data processing device provided in this embodiment can abstract and integrate rich information such as domain models, business data, and system resources into a single system for data management. This reduces the difficulty of data modeling and the overhead of synchronizing metadata information between different engines to achieve the same metadata in the data map system. The server uses a unified graph processing engine and exposes a unified graph interface, further reducing the complexity of the system. At the same time, due to the introduction of full-map, it becomes very natural to realize complex business requirements such as data lineage, and the development difficulty is greatly reduced.

[0093] In some implementations, the modeling processing module 52 described above is further configured to: acquire initial data resources corresponding to one or more external physical engines; wherein the initial data resources include at least one or more of physical resources, data sources, data models, data tags, data processing tasks, business resources, and management resources; perform data transformation processing on the initial data resources to obtain graph resources and graph relationships; and determine the target domain model through graph resources and graph relationships.

[0094] In some embodiments, the modeling processing module 52 described above is further configured to: clean the initial data resources, and extract data entities and map entity relationships on the cleaned data to obtain graph resources and graph relationships; wherein, graph resources are used to represent the data entities in the initial resources, and graph relationships are used to represent the relationships between data entities in a graphical form.

[0095] In some implementations, the target domain model includes at least one or more of the following models: a first graph model obtained by extracting entities and relationships from a data model; a second graph model obtained by extracting entities and relationships from a data model and data labels; a third graph model obtained by extracting entities and relationships from data processing tasks; a fourth graph model obtained by extracting entities and relationships from a data source and a data model; a fifth graph model obtained by extracting entities and relationships from a data source and physical resources; and a sixth graph model obtained by extracting entities and relationships from business resources, data models, and management resources.

[0096] In some embodiments, the above-described apparatus further includes: a classification and search module, configured to: classify the graph resources to obtain the classification category to which the graph resources belong; wherein the graph resources correspond to at least one category; and, in response to input for the classification category, search for graph resources in the target domain model and match corresponding graph relationships based on the searched graph resources.

[0097] In some embodiments, the above apparatus further includes a creation module for: creating new data resources in response to the creation operation of the graph engine; classifying the new data resources; and matching the new data resources to the corresponding graph relationships in the target domain model based on the classification attributes.

[0098] In some implementations, the external physical engine includes the Hive engine, and the graph resource includes Hive data tables; the creation module is further configured to: create new graph resources in response to a creation operation on table information in the Hive data table; wherein the table information includes at least the table name, table fields, list information, field types, and field comments.

[0099] In some implementations, the data display module 56 is also used to: connect the specified API interface of the data map to an external system via a public API based on OpenAPI, so as to display the initial data resources and / or the business processing results of data business processing.

[0100] The full-map data processing device provided in this application embodiment has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the embodiment of the full-map data processing device can be referred to the corresponding content in the aforementioned full-map data processing method embodiment.

[0101] This application also provides an electronic device, such as... Figure 6 The diagram shows the structure of the electronic device 100, which includes a processor 61 and a memory 60. The memory 60 stores computer-executable instructions that can be executed by the processor 61. The processor 61 executes the computer-executable instructions to implement any of the above-mentioned full-map data map systems.

[0102] exist Figure 6 In the illustrated embodiment, the electronic device further includes a bus 62 and a communication interface 63, wherein the processor 61, the communication interface 63, and the memory 60 are connected via the bus 62.

[0103] The memory 60 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 63 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 62 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 62 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0104] Processor 61 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 61 or by instructions in software form. Processor 61 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory. The processor 61 reads the information in the memory and, in conjunction with its hardware, completes the steps of the full-map data map system described in the foregoing embodiment.

[0105] This application also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the aforementioned full-map data map system. For specific implementation details, please refer to the foregoing method embodiments, which will not be repeated here.

[0106] The computer program products of the full-map data processing method, apparatus, device and storage medium provided in the embodiments of this application include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0107] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application.

[0108] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0109] In the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0110] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for processing full-map data, characterized in that, The method includes: Acquire initial data resources from one or more external physical engines; wherein, the initial data resources include at least one or more of physical resources, data sources, data models, data tags, data processing tasks, business resources, and management resources; The initial data resources are cleaned, and data entities are extracted and entity relationships are mapped from the cleaned data to obtain graph resources and graph relationships. The graph resources are used to represent the data entities in the initial data resources, and the graph relationships are used to represent the relationships between the data entities in a graphical form. The target domain model is represented by the graph resources and graph relationships; data business processing is performed based on the graph resources and graph relationships in the target domain model. Based on OpenAPI, the initial data resources and / or the business processing results of data business processing are displayed. The method further includes: In response to the creation operation of the graph engine, create new data resources; The newly added data resources are classified and processed, and the newly added data resources are matched to the corresponding graph relationships in the target domain model based on the classification attributes.

2. The full-map data processing method according to claim 1, characterized in that, The target domain model includes at least one or more of the following models: a first graph model obtained by extracting entities and relationships from the data model; a second graph model obtained by extracting entities and relationships from the data model and the data labels; a third graph model obtained by extracting entities and relationships from the data processing task; a fourth graph model obtained by extracting entities and relationships from the data source and the data model; a fifth graph model obtained by extracting entities and relationships from the data source and the physical resources; and a sixth graph model obtained by extracting entities and relationships from the business resources, the data model, and the management resources.

3. The full-map data processing method according to claim 1, characterized in that, The method further includes: The graphical resources are classified to obtain the classification category to which the graphical resources belong; wherein the graphical resources correspond to at least one category; In response to input for the classification category, the graph resources in the target domain model are searched, and the corresponding graph relationships are matched based on the searched graph resources.

4. The full-map data processing method according to claim 1, characterized in that, The external physical engine includes the Hive engine, and the graph resources include Hive data tables; In response to the graph engine's creation operation, new data resources are created, including: In response to the creation operation of table information in the Hive data table, a new graph resource is created; wherein the table information includes at least the table name, table fields, list information, field types, and field comments.

5. The full-map data processing method according to claim 4, characterized in that, Based on the OpenAPI, the initial data resources and / or the business processing results of data business processing are displayed, including: Based on the OpenAPI, the specified API interface of the data map is connected to external systems through a public API to display the initial data resources and / or the business processing results of data business processing.

6. A full-map data processing device, characterized in that, The device includes: A modeling processing module is used to acquire initial data resources from one or more external physical engines; wherein, the initial data resources include at least one or more of physical resources, data sources, data models, data tags, data processing tasks, business resources, and management resources; The initial data resources are cleaned, and data entities are extracted and entity relationships are mapped from the cleaned data to obtain graph resources and graph relationships. The graph resources are used to represent the data entities in the initial data resources, and the graph relationships are used to represent the relationships between the data entities in a graphical form. The target domain model is represented by the graph resources and graph relationships; The data service processing module is used to perform data service processing based on the graph resources and graph relationships in the target domain model; The data display module is used to display the initial data resources and / or the business processing results of data business processing based on OpenAPI; The device further includes a creation module, wherein the creation module is used for: In response to the creation operation of the graph engine, create new data resources; The newly added data resources are classified and processed, and the newly added data resources are matched to the corresponding graph relationships in the target domain model based on the classification attributes.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the full-map data processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the full-map data processing method according to any one of claims 1 to 5.

Images