Distributed data virtual fusion method and system based on cloud edge collaboration mechanism

Abstract

The invention belongs to the technical field of big data, and particularly discloses a distributed data virtual fusion method and system based on a cloud edge collaboration mechanism, and the method comprises the steps: obtaining a plurality of data sources at an edge end, and carrying out the adaptive connection of the plurality of data sources; wherein each data source comprises a plurality of data source tables corresponding to the data source; selecting a required data source table from the plurality of data sources which are adaptively connected to import the data source table; obtaining metadata information of the imported data source table, and carrying out storage and organization management; constructing a unified virtual data model, and dynamically mapping the physical data model in the imported data source table to the unified virtual data model to construct a virtual table; for the business themes, the virtual tables belonging to the unified business themes are associated and combined, and a virtual data warehouse is constructed; and publishing the unified virtual data model and the virtual data warehouse in a service form for a data consumer to call, thereby obtaining physical data information corresponding to the metadata information.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data, and particularly relates to a distributed data virtual fusion method and system based on a cloud-edge collaboration mechanism. Background Art

[0002] Big data is the basis for realizing intelligent applications and empowering the digital transformation of organizations. The full sharing and efficient transfer of big data are important guarantees for achieving in-depth integration of cross-departmental operations. Currently, data within an organization is continuously generated, and the amount of data is increasing day by day, with characteristics such as wide distribution of objects, multiple types, cross-levels, cross-platforms, and heterogeneous heterogeneity. However, due to obstacles in aspects such as administrative management and information technology, serious information silos exist, and the phenomena of "unwilling to share" and "unable to share" of source data are prominent. Data cannot enter the data collection, processing, fusion, and application link in a timely manner, greatly restricting data applications. Therefore, how to build an agile and dynamic data fusion architecture, use efficient means to achieve the collection, governance, and shared application of massive data, and realize the analysis, mining, and prediction of the overall situation, so as to support cross-departmental data fusion applications is an urgent problem to be solved. Summary of the Invention

[0003] The present invention provides a distributed data virtual fusion method and system based on a cloud-edge collaboration mechanism to solve the problems that when facing the convergence and fusion of widely distributed data sources, it will result in high bandwidth consumption, and the design and maintenance complexity of a large number of data platform tasks is relatively high. The centralized storage of massive data requires high hardware investment costs, and the multiple replication and movement of data will also cause data security to be uncontrollable.

[0004] In a first aspect, a distributed data virtual fusion method based on a cloud-edge collaboration mechanism is provided. The method includes:

[0005] Obtain a number of data sources at the edge side and adaptively connect the number of data sources;

[0006] Wherein, each data source includes a number of data source tables corresponding to itself;

[0007] Select the required data source tables from the number of adaptively connected data sources for import;

[0008] Obtain the metadata information of the imported data source tables, and perform storage and organizational management;

[0009] Build a unified virtual data model, and dynamically map the physical data model in the imported data source tables to the unified virtual data model to build virtual tables;

[0010] For a business theme, associate and combine the virtual tables belonging to the unified business theme to build a virtual data warehouse;

[0011] Publish the unified virtual data model and virtual data warehouse in the form of services for data consumers to call, so as to obtain the physical data information corresponding to the metadata information.

[0012] In a second aspect, a distributed data virtual fusion system based on a cloud-edge collaboration mechanism is provided. The system includes:

[0013] An edge data adaptation module, configured to obtain a plurality of data sources at the edge and adaptively connect the plurality of data sources; wherein each data source includes a plurality of data source tables corresponding to itself.

[0014] A data source table import module, configured to select and import the required data source tables from the plurality of adaptively connected data sources.

[0015] A metadata organization module, configured to obtain the metadata information of the imported data source tables and perform storage and organizational management.

[0016] A dynamic mapping module, configured to construct a unified virtual data model and dynamically map the physical data model in the imported data source tables to the unified virtual data model to construct virtual tables.

[0017] A virtual data warehouse construction module, configured to face business themes and associate and combine the virtual tables belonging to the unified business theme to construct a virtual data warehouse.

[0018] A data service call module, configured to publish the unified virtual data model and virtual data warehouse in the form of services for data consumers to call, so as to obtain the physical data information corresponding to the metadata information.

[0019] A distributed data virtual fusion method and system based on a cloud-edge collaboration mechanism provided by an embodiment of the present invention change the traditional "data replication" to "data connection", eliminate the security risks in the traditional data movement process, eliminate the data replication and storage costs, save 50%-80% of the time compared with the traditional method, and can effectively reduce the maintenance cost. A dynamic virtual data warehouse is established based on the virtual data resource pool, and data resource access permissions are uniformly granted to users.

[0020] Other features and advantages of the present invention will be described in the following specification, and part of them will become obvious from the specification or be understood by implementing the present invention. The objectives and other advantages of the present invention can be achieved and obtained through the structures specifically pointed out in the written specification, claims, and drawings. Description of the Drawings

[0021] The accompanying drawings described herein are used to provide a further understanding of the present invention and form a part of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0022] Figure 1 It is a schematic diagram of the implementation process of a distributed data virtual fusion method based on a cloud-edge collaboration mechanism provided according to an embodiment of the present invention. Specific embodiments

[0023] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this specification.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product, or equipment that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or equipment.

[0025] The following will detail the technical solutions provided by each embodiment of this specification in conjunction with the drawings.

[0026] To improve the sharing ability of big data, an embodiment of the present invention provides a distributed data virtual fusion method and system based on a cloud-edge collaboration mechanism.

[0027] Figure 1 It is a schematic diagram of the implementation process of a distributed data virtual fusion method based on a cloud-edge collaboration mechanism provided according to an embodiment of the present invention. Refer to Figure 1 , the method includes:

[0028] Obtain a number of data sources at the edge side and adaptively connect the number of data sources;

[0029] Among them, each data source includes a number of data source tables corresponding to itself;

[0030] Select the required data source tables from several adaptively connected data sources for import;

[0031] In this step, in specific implementation, for example, select N1 data source tables for import from data source 1, select N2 data source tables for import from data source 2,..., and select NN data source tables for import from data source N.

[0032] Obtain the metadata information of the imported data source tables, and store and organize them;

[0033] Construct a unified virtual data model, and dynamically map the physical data model in the imported data source tables to the unified virtual data model to construct virtual tables;

[0034] Face the business theme, associate and combine the virtual tables to construct a virtual data warehouse;

[0035] Publish the unified virtual data model and the virtual data warehouse in the form of services for data consumers to call, so as to obtain the stored and organized metadata information.

[0036] In a specific implementation manner, the obtaining of several data sources at the edge end and the adaptive connection of the several data sources specifically include:

[0037] Construct an edge - end data adapter, fill in the jdbcurl (Java database connection URL) connection information of the data source and the user password for authentication, realize the connection and adaptation of distributed data sources, provide unified access to data, and obtain and transmit the metadata of the edge - end data sources through ODBC (Open Database Connectivity) / JDBC (Java Database Connectivity), JSON (JavaScript Object Notation), or API (Application Programming Interface).

[0038] In a specific implementation manner, the metadata information includes the network location of the server where the data source table is located, the information for logging in to the database server, the name of the data source table, the owner, the date when the data source table was established, the structure of the data source table, the primary key and foreign key of the data source table, the number of rows of the data source table, and the values assigned to each column;

[0039] The obtaining of the metadata information of the imported data source tables and the storage and organization management specifically include:

[0040] When the data source tables are imported, obtain the metadata information of the data source tables opened at the data source end for registration;

[0041] Store and hierarchically organize the registered metadata information based on a tree - shaped index structure;

[0042] Among them, the root node in the tree structure represents any data source node, and the leaf node represents the attribute information of the data source table in the current data source. The attributes include the registered metadata information of the data table.

[0043] In a specific implementation, the steps of constructing a unified virtual data model and dynamically mapping the physical data model in the imported data source table to the unified virtual data model to construct a virtual table specifically include:

[0044] Establish a unified business domain metadata standard based on business application specifications;

[0045] Among them, the metadata standard includes field name, field comment, field type, field length, field precision, and whether it is null attribute information;

[0046] Follow the unified business domain metadata standard, and select necessary business domain metadata according to the needs of data theme aggregation or analysis applications to construct a unified virtual data model;

[0047] Among them, the unified virtual data model includes model name, model description, and theme data structure information. The theme data structure information includes field name, field comment, field type, field length, field precision, whether it is null, whether it is a primary key, and whether it is a foreign key attribute information;

[0048] Perform row selection (i.e., filtering the required data records according to rules), column selection (i.e., selecting the fields to be mapped), column connection (i.e., associating the data source table fields with the virtual data model fields), and conversion (including but not limited to field type conversion, field value calculation, multi-field fusion, etc.), column name and table name changes (such as setting field aliases, adding unified identifiers to the virtual data model fields), and grouped dynamic mapping operations (synchronous mapping and asynchronous mapping according to field grouping. Synchronous mapping means that there is a certain semantic correlation between the data source table fields and the virtual data model fields, and the system can intelligently identify the field relationship between the two and automatically establish a mapping relationship; asynchronous mapping means that there is no necessary semantic correlation between the two fields, or one data source table field needs to be mapped to multiple different virtual data model fields, or multiple data source table fields are fused and mapped to the same virtual data model field and other special cases, which need to be processed through special SQL scripts or programming languages) on the physical data model (i.e., database table schema) in the imported data source table to the unified virtual data model in the cloud, establish the association relationship between the physical data model at the data source end and the virtual data model in the cloud, and form a unified virtual data view corresponding to the accessed data source table.

[0049] In a specific embodiment, the unified virtual data model and the virtual data warehouse are published in the form of services for data consumers to call, so as to obtain the metadata information stored, organized and managed. Specifically, it includes:

[0050] View the unified virtual data model and the virtual data warehouse in the form of virtual tables, and provide them to data consumers through multiple interfaces. Retrieve and view the service details through the data service catalog;

[0051] Data consumers access the virtual tables through the Structured Query Language (SQL), actively pull data by applying according to the parameters defined by the interface, or automatically obtain the required data by subscribing. The call of the virtual table is transformed into the call of the physical data table at the data source end, so as to obtain the metadata information stored, organized and managed.

[0052] To better understand the present invention, the implementation process of the present invention will be described below in conjunction with specific embodiments.

[0053] The specific steps of the distributed data virtual fusion method based on the cloud-edge collaboration mechanism provided by the embodiments of the present invention include:

[0054] (1) Edge-side data adaptation connection

[0055] The edge side refers to the accessible network environment between the network domain where the user-side system is located and the network domain where the data source is located. If the networks of the two are logically isolated, the edge side generally refers to the front-end machine deployed in the network domain of the user-side system, and the data source network domain can be accessed through the front-end machine; if the two networks can be directly connected, the edge side can be any node on the network, and this node can connect the user-side system and the data source at the same time.

[0056] Edge-side data adaptation connection means connecting to the data source system (database or data service interface) through the edge-side front-end machine (or network node server) to be able to access the corresponding data tables. Edge-side data adaptation connection requires creating a new data source node, which includes data source information and data source connection information. Among them, the data source information includes information such as the data source name, database type, and version, and the data source connection information includes the jdbcurl connection information of the data source and identity authentication information such as the user name and password. When creating a new data source node, the above information needs to be filled in and a connectivity test is required.

[0057] (2) Access the data source tables

[0058] After the connectivity test passes, the database table information of the edge-side data source node can be viewed, and the range of data tables to be accessed or imported can be selected from them. The selected data source tables are used for the next step of data organization and model mapping. Operations such as adding, modifying, and deleting can be performed on the selected data source tables.

[0059] (3) Metadata Organization

[0060] While selecting the data source table, the metadata information of the source table opened at the data source end can be viewed. Based on the source table metadata information, register the data source table in the system. The registered metadata information includes the network location of the server where the source table is located (referring to the network domain and IP address where the data source end is located), the information for logging in to the database server (referring to the jdbc URL connection information, username, password, etc. for logging in to the database), the source table name, the owner, the source table creation date, the source table structure (including field name, field comment, field type, field length, field precision, non-null specification, etc.), the primary key and foreign key defined by the source table, the number of rows in the source table (referring to the number of data records), and the values assigned to each column (for query optimization).

[0061] Hierarchically organize and manage the metadata based on the tree index structure. The root node in the tree structure is the current data source node, and the leaf nodes are the selected data tables under this data source. Clicking on the leaf nodes can further expand to view the registered metadata information of the data source tables.

[0062] (4) Establish a Unified Virtual Data Model

[0063] Based on the business application specifications, establish a unified business domain metadata standard. The metadata standard includes attribute information such as field name, field comment, field type, field length, field precision, and whether it is null.

[0064] Following the unified business domain metadata standard, according to the needs of data theme aggregation or analysis applications, select the necessary business domain metadata or in the form of SQL scripts to construct a unified virtual data model. The unified virtual data model includes the model name, model description, and complete theme data structure information. The theme data structure information includes attribute information such as field name, field comment, field type, field length, field precision, whether it is null, whether it is the primary key, and whether it is the foreign key. View the already created unified virtual data model in list form and perform maintenance operations such as addition, deletion, modification, and query on the unified virtual data model.

[0065] (5) Dynamic Data Mapping

[0066] Dynamically map the physical data model in the finally registered data source table and the established unified virtual data model by fields, construct a real-time association relationship between the source-side physical data model and the system-side virtual data model, and form a unified virtual table corresponding to the accessed data source table.

[0067] The mapping operations include (note: those not marked as optional are mandatory operations):

[0068] a. Row selection, which supports filtering the required data records according to specific rules. This operation is optional;

[0069] b. Column selection, which supports selecting the fields to be mapped in manual or automatic matching mode;

[0070] c. Column connection, which supports establishing an association relationship between the data source table fields and the virtual data model fields in manual or automatic matching mode;

[0071] d. Column conversion, which supports data processing such as field type conversion, field value calculation, and multi-field fusion through SQL scripts. This operation is optional;

[0072] e. Changes to column names and table names, which support setting field aliases, adding unified identifiers to virtual data model fields, etc., to enhance the adaptability of model applications. This operation is optional;

[0073] f. Grouping dynamic mapping operation, which supports synchronous mapping and asynchronous mapping according to field grouping. In the synchronous mapping process, for the data source table fields and the virtual data model fields with a certain semantic correlation, the system can intelligently identify the field relationship between the two and automatically establish a mapping relationship; in the asynchronous mapping process, for special cases where there is no necessary semantic correlation between the two fields, or one field in the data source table needs to be mapped to multiple different fields in the virtual data model, or multiple fields in the data source table are fused and mapped to the same field in the virtual data model, etc., special SQL scripts or programming languages are used for processing to achieve the field mapping between the data source table and the virtual data model. One of the synchronous mapping operation and the asynchronous mapping operation must be selected, or both can be executed simultaneously.

[0074] (6) Construct a virtual data warehouse

[0075] Organize and fuse the virtual data model mapped in step (5) again according to the business theme or specific application scenario requirements, and perform operations such as data query, filtering, aggregation on a single model or association, combination, field fusion, and adding field information between multiple models using SQL scripts or programming languages to form new business-oriented virtual tables. These virtual data tables are saved to form a virtual data warehouse for special topic data analysis.

[0076] (7) Data service publishing

[0077] The virtual data model and virtual data view can both be viewed in the form of virtual tables and provided to data consumers through multiple interfaces such as JDBC and SOAP. The service details can be retrieved and viewed through the data service directory. Before the data service is published, it is invisible to service consumers. It must be "publicly published" (visible to all) or "privately published" (partially visible, associated with corresponding data consumers) before data consumers can see it.

[0078] (8) Data service call

[0079] Data consumers can access the virtual data table through SQL, actively pull data by applying according to the parameters defined in the interface, or automatically obtain the required data through subscription. The call to the virtual data table will be further transformed into a call to the physical data table at the source end. There is an access control mechanism during the data service call process to verify that the data consumer can use the corresponding data before the service call is made, ensuring that the data consumer can only access the data services within the permitted scope.

[0080] A distributed data virtual fusion method based on a cloud-edge collaboration mechanism provided by an embodiment of the present invention relies on advanced technologies such as data virtualization and edge computing to build a cross-departmental data aggregation and fusion system with cloud-edge collaboration, realizing seamless access and virtual aggregation and fusion of cross-departmental multi-source heterogeneous data in the mode where the source-end data does not move, forming a new data resource system that can reflect real-time changes in source-end data, aggregate cross-departmental data on demand, and "delete after use", quickly responding to various data query, access, analysis, and calculation requirements, and providing business data fusion services that can be shared, authorized, and used on demand for various business applications, effectively protecting the source-end data from leaving the local domain and improving the security and timeliness of cross-departmental data use.

[0081] Based on the same inventive concept, the present invention also provides a distributed data virtual fusion system based on a cloud-edge collaboration mechanism, and the system includes:

[0082] An edge data adaptation module, configured to obtain a plurality of data sources at the edge end and adaptively connect the plurality of data sources; wherein each data source includes a corresponding plurality of data source tables.

[0083] A data source table import module, configured to select and import the required data source tables from the plurality of data sources that have been adaptively connected.

[0084] A metadata organization module, configured to obtain the metadata information of the imported data source tables, and store and organize and manage them.

[0085] A dynamic mapping module, configured to build a unified virtual data model, and dynamically map the physical data model in the imported data source tables to the unified virtual data model to build virtual tables.

[0086] A virtual data warehouse construction module, configured to, for a business theme, associate and combine the virtual tables belonging to the unified business theme to build a virtual data warehouse.

[0087] A data service call module, which is used to publish the unified virtual data model and the virtual data warehouse in the form of services for data consumers to call, and then obtain the physical data information corresponding to the metadata information.

[0088] In a specific implementation manner, the edge data adaptation module is specifically used for:

[0089] Construct an edge-side data adapter, fill in the jdbcurl connection information of the data source and the user password for authentication, implement the connection and adaptation of the distributed data source, provide unified access to the data, and obtain and transmit the metadata of the edge-side data source through ODBC / JDBC, JSON or API.

[0090] In a specific implementation manner, the metadata information includes the network location of the server where the data source table is located, the information for logging in to the database server, the name of the data source table, the owner, the date when the data source table was established, the structure of the data source table, the primary key and foreign key of the data source table, the number of rows in the data source table, and the values assigned to each column;

[0091] The metadata organization module is specifically used for:

[0092] When importing the data source table, obtain the metadata information of the data source table opened at the data source end for registration;

[0093] Store and hierarchically organize and manage the registered metadata information based on a tree index structure;

[0094] Among them, the root node in the tree structure represents any data source, and the leaf node represents the attribute information of the data source table in the current data source.

[0095] In a specific implementation manner, the dynamic mapping module is specifically used for:

[0096] Establish a unified business domain metadata standard based on business application specifications;

[0097] Among them, the metadata standard includes field name, field comment, field type, field length, field precision, and whether it is null attribute information;

[0098] Follow the unified business domain metadata standard, and select the necessary business domain metadata according to the needs of data theme aggregation or analysis applications to construct a unified virtual data model;

[0099] Among them, the unified virtual data model includes a model name, a model description, and theme data structure information. The theme data structure information includes field name, field comment, field type, field length, field precision, whether it is null, whether it is the primary key, and whether it is the foreign key attribute information;

[0100] Perform dynamic mapping operations for row selection to filter required data records according to rules by field from the physical data model in the imported data source table to the unified virtual data model in the cloud, dynamic mapping operations for column selection to select fields to be mapped, dynamic mapping operations for column connection to associate the fields of the data source table with the fields of the virtual data model, dynamic mapping operations for field type conversion, dynamic mapping operations for changes in column names and table names, and group dynamic mapping operations for synchronous mapping and asynchronous mapping by field grouping;

[0101] Among them, synchronous mapping means that there is a semantic correlation between the fields of the data source table and the virtual data model, and the system can intelligently identify the field relationship between the two and automatically establish a mapping relationship; asynchronous mapping means that there is no semantic correlation between the two fields, or one field of the data source table needs to be mapped to multiple different fields of the virtual data model, or in special cases where multiple fields of the data source table are fused and mapped to the same field of the virtual data model, it is processed through a dedicated SQL script or programming language to establish the association relationship between the physical data model at the data source end and the virtual data model in the cloud, forming a unified virtual data view corresponding to the accessed data source table.

[0102] In a specific embodiment, the data service invocation module is specifically used for:

[0103] View the unified virtual data model and the virtual data warehouse in the form of virtual tables, and provide them to data consumers through multiple interfaces, and retrieve and view service details through the data service catalog;

[0104] Data consumers access the virtual tables through the Structured Query Language (SQL), actively pull data by applying according to the parameters defined by the interface, or automatically obtain the required data by subscribing. The invocation of the virtual tables is transformed into the invocation of the physical data tables at the data source end, and then the metadata information stored and organized is obtained.

[0105] To better understand the present invention, the implementation process of the present invention will be described below in conjunction with specific embodiments.

[0106] The distributed data virtual fusion system based on the cloud-edge collaboration mechanism provided by the embodiment of the present invention includes:

[0107] (1) Construct an edge data adapter

[0108] Construct an edge data end data adapter to realize the connection and adaptation of distributed data sources, shield the access details of underlying heterogeneous data sources, provide unified access to data, and realize the acquisition and transmission of metadata of edge data nodes and the delivery of the final upper-layer data resources through ODBC / JDBC, JSON, API, etc. The edge data nodes can open all or part of the metadata views to the data virtualization layer according to their own policies.

[0109] (2) Virtual Convergence and Fusion of Cloud Data

[0110] The core idea of the virtual convergence and fusion of cloud data is to replace data integration with metadata integration and replace data physical movement with data virtual mapping, thereby effectively reducing the costs brought by the movement, replication, and storage of a large amount of data.

[0111] Data Mapping Function

[0112] The dynamic mapping module solves the problem of virtual access to distributed data sources. This module mainly conducts connection management for heterogeneous data sources (such as Oracle, MySQL, PostgreSQL, SQL Server, DM, etc.), imports, encapsulates, and publishes source tables, then constructs a unified virtual data model according to data standards, and performs layer-by-layer mapping based on business logic on the unified virtual data model, finally forming a virtual data warehouse oriented to business themes.

[0113] Data Source Registration: Register the metadata information of the data source, including the server address information for storing the source table, the connection information for logging in to the database, the name, owner, creation date of the source table, and the description information of each data element in the table, etc.

[0114] Data Source Mapping: Establish a unified virtual data model, construct virtual data tables based on unified data standards, and realize the conversion of the physical data model of the source table to the virtual data model. For example, combine the data in multiple source tables together, or convert specific column values, etc.

[0115] Metadata Organization Function

[0116] In the implementation process of the present invention, the physical data of the data source is not stored, but the metadata of different data sources is organized, stored, and managed, and is used as the basis for defining virtual data views for the data service layer. Adopting this method can greatly save physical space and complete data sharing operations without physical data concentration.

[0117] For the query requests of users, high-performance metadata organization, storage, and fast search are the keys to ensuring that users obtain the required data resources. The metadata organization unit realizes the extraction and storage of metadata, and analyzes and re-classifies the metadata according to the user's query requirements and the relationships between metadata, establishes a general metadata organization model with good structure and relevance, locates the storage location of the corresponding data through metadata information, and then uses interfaces such as ODBC / JDBC, API, JSON, etc. to obtain data information, thereby improving the query efficiency.

[0118] Data Service Function

[0119] The data service unit encapsulates the virtual data model, provides a unified data service for users, builds a unified data service portal, provides a unified data service export, helps organizations enhance the value of data assets, and at the same time ensures the reliability, security and effectiveness of data. It supports building data service interfaces through interface configuration to meet different data usage scenarios, while reducing the development threshold of data and helping organizations maximize the value of data applications.

[0120] (3) Cloud-edge collaborative data access and invocation

[0121] Build an application model for big data scenarios at the application layer, transfer the data query requests of the application model to the data unit, and call relevant services according to the data service interface parameters. Provide multiple access interfaces for the query requests of data consumers. For example, some data consumers can access using JDBC / ODBC interfaces, or access the same data service based on the Simple Object Access Protocol (SOAP) interface. Data consumers can initiate query requests according to their determined access methods.

[0122] A distributed data virtual fusion system based on the cloud-edge collaborative mechanism provided by an embodiment of the present invention addresses the practical challenge that large-scale cross-department data cannot be physically aggregated under data security constraints. Deploy data conversion models distributively at the edge to achieve real-time dynamic mapping from the physical data model to the unified virtual data model, realize the real-time conversion from the virtual data model to the physical data model, and at the same time perform operations such as combination and association on the virtual data model for business themes, converge them into a virtual data warehouse, build a global data virtual view, shield the complexity of the source-side data, and publish the data in the form of services for data consumers to call, so as to complete the global data virtual aggregation, governance and fusion without moving the source-side physical data.

[0123] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they know the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the present invention.

[0124] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these changes and modifications.

Claims

1. A distributed data virtual fusion method based on cloud-edge collaboration mechanism, characterized in that: The method comprises: Acquire several data sources at the edge end, and adapt and connect the several data sources; Wherein, each data source includes several data source tables corresponding to itself; Select the required data source table from several data sources that have been adapted and connected for import; Obtain metadata information of the imported data source table, and store and organize it; Build a unified virtual data model, and dynamically map the physical data model in the imported data source table to the unified virtual data model to build a virtual table; Focusing on business themes, virtual tables belonging to the same business theme are associated and combined to build a virtual data warehouse; The unified virtual data model and virtual data warehouse are published in the form of services for data consumers to call and obtain the physical data information corresponding to the metadata information.

2. The method according to claim 1, characterized in that The step of obtaining a plurality of data sources at the edge end and adapting and connecting the plurality of data sources specifically includes: Construct an edge data adapter, fill in the jdbcurl connection information and user password of the data source for authentication, realize the connection and adaptation of distributed data sources, provide unified access to data, and realize the acquisition and transmission of edge data source metadata through ODBC / JDBC, JSON or API.

3. The method according to claim 2, characterized in that The metadata information includes the network location of the server where the data source table is located, the information for logging into the database server, the name of the data source table, the owner and the date when the data source table was created, the structure of the data source table, the primary key and foreign keys of the data source table, the number of rows in the data source table, and the values ​​assigned to each column; The metadata information of the imported data source table is obtained and stored and organized, specifically including: While the data source table is being imported, the metadata information of the data source table opened on the data source end is obtained for registration; Store and hierarchically organize the registered metadata information based on a tree index structure; The root node in the tree structure represents any data source node, and the leaf nodes represent the attribute information of the data source table in the current data source, wherein the attributes include the registered data table meta-table information.

4. The method according to claim 3, characterized in that: The step of constructing a unified virtual data model and dynamically mapping the physical data model in the imported data source table to the unified virtual data model to construct a virtual table specifically includes: Establish unified business domain metadata standards based on business application specifications; The metadata standard includes field name, field annotation, field type, field length, field precision and whether it is empty attribute information; Following the unified business domain metadata standard, according to the data subject aggregation or analysis application needs, necessary business domain metadata is selected to build a unified virtual data model; The unified virtual data model includes a model name, a model description and subject data structure information, and the subject data structure information includes field name, field comment, field type, field length, field precision, whether it is empty, whether it is a primary key and whether it is a foreign key attribute information; The physical data model imported into the data source table is mapped to the unified virtual data model on the cloud by fields, and the row selection dynamic mapping operation is performed to filter the required data records according to the rules, the column selection dynamic mapping operation is performed to select the fields to be mapped, the column connection dynamic mapping operation is performed to associate the data source table fields with the virtual data model fields, the field type conversion dynamic mapping operation, the column name and table name change dynamic mapping operation, and the group dynamic mapping operation is performed by field grouping for synchronous mapping and asynchronous mapping; Among them, synchronous mapping means that there is semantic correlation between the fields of the data source table and the fields of the virtual data model, and the system can intelligently identify the relationship between the two fields and automatically establish a mapping relationship; asynchronous mapping means that there is no semantic correlation between the two fields, or a field of the data source table needs to map multiple different fields of the virtual data model, or multiple fields of the data source table are merged and mapped to the same virtual data model field. In special cases, special SQL scripts or programming languages ​​are used for processing to establish the association relationship between the physical data model on the data source side and the virtual data model on the cloud, forming a unified virtual data view corresponding to the access data source table.

5. The method according to claim 4, characterized in that The unified virtual data model and virtual data warehouse are published in the form of services for data consumers to call and obtain metadata information for storage and organization management, including: The unified virtual data model and virtual data warehouse are viewed in the form of virtual tables and provided to data consumers through various interfaces, and service details are retrieved and viewed through the data service catalog; Data consumers access virtual tables through the structured query language SQL, actively pull data through applications based on the parameters defined in the interface, or automatically obtain required data through subscriptions. Calls to virtual tables are converted into calls to physical data tables on the data source side, thereby obtaining metadata information for storage and organizational management.

6. A distributed data virtual fusion system based on cloud-edge collaboration mechanism, characterized in that: The distributed data virtual fusion method based on the cloud-edge collaboration mechanism according to any one of claims 1 to 5 is adopted, and the system includes: The edge data adaptation module is used to obtain several data sources at the edge end and adapt and connect the several data sources; wherein each data source includes several data source tables corresponding to itself; The data source table import module is used to select the required data source table from a number of data sources that have been adapted and connected for import; The metadata organization module is used to obtain metadata information of the imported data source table and store and organize it; A dynamic mapping module is used to construct a unified virtual data model and dynamically map the physical data model in the imported data source table to the unified virtual data model to construct a virtual table; The virtual data warehouse construction module is used to build a virtual data warehouse by associating and combining virtual tables belonging to the same business subject. The data service calling module is used to publish the unified virtual data model and virtual data warehouse in the form of services for data consumers to call, and then obtain the physical data information corresponding to the metadata information.

7. The system according to claim 6, characterized in that The edge data adaptation module is specifically used for: Construct an edge data adapter, fill in the jdbcurl connection information and user password of the data source for authentication, realize the connection and adaptation of distributed data sources, provide unified access to data, and realize the acquisition and transmission of edge data source metadata through ODBC / JDBC, JSON or API.

8. The system according to claim 7, characterized in that The metadata information includes the network location of the server where the data source table is located, the information for logging into the database server, the name of the data source table, the owner and the date when the data source table was created, the structure of the data source table, the primary key and foreign keys of the data source table, the number of rows in the data source table, and the values ​​assigned to each column; The metadata organization module is specifically used for: While the data source table is being imported, the metadata information of the data source table opened on the data source end is obtained for registration; Store and hierarchically organize the registered metadata information based on a tree index structure; The root node in the tree structure represents any data source node, and the leaf nodes represent the attribute information of the data source table in the current data source, wherein the attributes include the registered data table meta-table information.

9. The system according to claim 8, characterized in that The dynamic mapping module is specifically used for: Establish unified business domain metadata standards based on business application specifications; The metadata standard includes field name, field annotation, field type, field length, field precision and whether it is empty attribute information; Following the unified business domain metadata standard, according to the data subject aggregation or analysis application needs, necessary business domain metadata is selected to build a unified virtual data model; The unified virtual data model includes a model name, a model description and subject data structure information, and the subject data structure information includes field name, field comment, field type, field length, field precision, whether it is empty, whether it is a primary key and whether it is a foreign key attribute information; The physical data model imported into the data source table is mapped to the unified virtual data model on the cloud by fields, and the row selection dynamic mapping operation is performed to filter the required data records according to the rules, the column selection dynamic mapping operation is performed to select the fields to be mapped, the column connection dynamic mapping operation is performed to associate the data source table fields with the virtual data model fields, the field type conversion dynamic mapping operation, the column name and table name change dynamic mapping operation, and the group dynamic mapping operation is performed by field grouping for synchronous mapping and asynchronous mapping; Among them, synchronous mapping means that there is semantic correlation between the fields of the data source table and the fields of the virtual data model, and the system can intelligently identify the relationship between the two fields and automatically establish a mapping relationship; asynchronous mapping means that there is no semantic correlation between the two fields, or a field of the data source table needs to map multiple different fields of the virtual data model, or multiple fields of the data source table are merged and mapped to the same virtual data model field. In special cases, special SQL scripts or programming languages ​​are used for processing to establish the association relationship between the physical data model on the data source side and the virtual data model on the cloud, forming a unified virtual data view corresponding to the access data source table.

10. The system according to claim 9, characterized in that The data service calling module is specifically used for: The unified virtual data model and virtual data warehouse are viewed in the form of virtual tables and provided to data consumers through various interfaces, and service details are retrieved and viewed through the data service catalog; Data consumers access virtual tables through the structured query language SQL, actively pull data through applications based on the parameters defined in the interface, or automatically obtain required data through subscriptions. Calls to virtual tables are converted into calls to physical data tables on the data source side, thereby obtaining metadata information for storage and organizational management.

Images