Data federation method, system, electronic device, and storage medium

Abstract

A data federation method, a system, an electronic device, and a storage medium. The data federation system comprises: a control plane (110) and a data plane (120), the data plane (120) being provided with two or more data layers (121), and the data layers (121) being respectively configured to connect different multi-source heterogeneous data systems. The control plane (110) comprises a gateway component (111) and a federation component (112), and each data layer (121) comprises a sidecar component (1211) and a semantic engine (1212). The gateway component (111) is configured to receive a data consumption request corresponding to N data layers, divide the data consumption request into N sub-requests, send same to the sidecar components (1211) of the corresponding data layers among the N data layers, respectively, receive second data returned by the sidecar components (1211) of the N data layers, send same to the federation component (112), and provide first data returned by the federation component (112) to a user. The sidecar components (1211) are configured to receive the sub-requests, then transfer same to the semantic engine (1212) of a current data layer, and return the second data returned by the semantic engine (1212) to the gateway component (111). The semantic engine (1212) is configured to query, on the basis of the sub-requests, source data from the multi-source heterogeneous data system connected to the current data layer and generate the second data on the basis of the source data. The federation component (112) is configured to fuse the second data of the N data layers to generate the first data, wherein N is an integer greater than 1. By means of a service grid technology-based data federation system, the query of service association between the source data in different fields can be implemented, so as to meet the use requirement of real-time change of a service requirement while reducing the costs.

Description

Data union method, system, electronic device and storage medium Technical Field

[0001] The present application relates to a data union method, system, electronic device and storage medium. Background Art

[0002] The management and construction of data assets is crucial. Currently, to achieve synergy across different value chains, big data platforms often use ETL tools to regularly synchronize data from heterogeneous systems into data warehouses, creating intelligent business systems for visualizing and analyzing data.

[0003] However, intelligent business systems built with ETL tools require customized data models based on user needs. Data is retrieved from data sources using fixed data extraction rules and stored in the data warehouse. Furthermore, data association and data extraction from newly added heterogeneous systems also require customized development. Changing user needs or adding new heterogeneous systems to the intelligent business system necessitates redevelopment or additional custom development, which is costly and lacks applicability, making it difficult to meet the actual needs of most enterprises for big data platforms.

[0004] Therefore, a new data federation solution is needed to reduce development costs and meet practical application needs.

[0005] Summary of the Invention

[0006] In view of this, the data union method, device, system, electronic device and storage medium provided in this application can meet the application requirements of real-time changes in business needs while reducing costs.

[0007] According to a first aspect of an embodiment of the present application, a data federation system is provided, comprising: a control plane and a data plane, wherein two or more data layers are configured in the data plane, each of the data layers being used to connect to a different multi-source heterogeneous data system; wherein the control plane includes a gateway component and a federation component, and each data layer includes a sidecar component and a semantic engine; the gateway component is configured to receive data consumption requests corresponding to N data layers, split the data consumption requests into N sub-requests, and send them respectively to the sidecar components of the corresponding data layers in the N data layers, receive second data returned by the sidecar components of the N data layers, transmit the second data to the federation component, and provide the first data returned by the federation component to a user; the sidecar component is configured to receive the sub-requests and transmit them to the semantic engine of the current data layer, and return the second data returned by the semantic engine to the gateway component; the semantic engine is configured to query the multi-source heterogeneous data systems connected to the current data layer for source data according to the sub-requests and generate the second data based on the source data; the federation component is configured to fuse the second data of the N data layers to generate the first data; wherein N is an integer greater than 1.

[0008] According to a second aspect of an embodiment of the present application, a data federation method is provided. The data federation method is implemented by the data federation system according to any one of claims 1 to 10, and the method includes:

[0009] The gateway component of the control plane receives data consumption requests corresponding to N data layers, splits the data consumption requests into N sub-requests according to the entity information in the data consumption requests, and distributes the sub-requests to the corresponding data layers according to the entity information in each sub-request, where each sub-request contains entity information of one data layer among all the entity information in the data consumption request;

[0010] After receiving the sub-request through the sidecar component, the N data layers respectively query the multi-source heterogeneous data system according to the entity information in the sub-request to generate second data, and return the second data to the control plane through the sidecar component, where the second data is used to indicate the business attributes and business associations of the entity identified by the entity information in the sub-request;

[0011] After receiving the second data of the N data layers through the gateway component, the control plane fuses the second data to generate first data, and provides the first data to the user through the gateway component, where the first data is used to indicate the service attributes and service associations of the entity identified by the entity information in the data consumption request;

[0012] Wherein, N is an integer greater than 1.

[0013] According to the third aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a bus, wherein the processor, the memory and the communication interface communicate with each other through the bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the data union method described in the first aspect above.

[0014] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes the data union method as described in the second aspect above.

[0015] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, which is tangibly stored on a computer-readable medium and includes computer-executable instructions, which, when executed, enable at least one processor to perform the data joining method described in the first aspect above.

[0016] It can be seen from the above technical solution that by dividing the data federation system into two parts, the control plane and the data plane, the data plane is configured with two or more data layers, each data layer is connected to multi-source heterogeneous data systems in different fields respectively, each data layer is configured with a sidecar component, and the control plane is configured with a gateway component. The communication between each data layer and the control plane can be realized through the connection between the sidecar component and the gateway component. The data plane is configured with a semantic engine, and the semantic engine can generate second data indicating the business association of its corresponding field. The control plane is configured with a federation component, and the federation component can generate first data that can indicate the business association of all fields by fusing the second data of each data layer. The data federation system based on service grid technology can realize the query of business associations between source data in various fields. It adopts a loosely coupled, highly robust and low-coding architecture with better flexibility and low development cost. It can support the flexible addition of multi-source heterogeneous data systems to meet changing application requirements, thereby meeting the application requirements of real-time changes in business requirements while reducing costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG1 is a schematic diagram of the architecture of a data federation system according to an embodiment of the present application;

[0018] FIG2 is a schematic diagram of an application scenario of a data federation system according to an embodiment of the present application;

[0019] FIG3 is a flow chart of a data federation method according to an embodiment of the present application;

[0020] FIG4 is a schematic diagram of a first knowledge graph obtained by fusing three second knowledge graphs according to an embodiment of the present application;

[0021] FIG5 is a flowchart of adding a multi-source heterogeneous data system according to an embodiment of the present application;

[0022] FIG6 is a schematic diagram of an electronic device according to an embodiment of the present application.

[0023] List of reference numerals: 100: data federation system 110: control plane 120: data plane 121: data layer 111: gateway component 112: federation component 113: discovery and registration component 114: authentication component 115: graph database 1211: sidecar component 1212: semantic engine 1213: binding component 1214: universal connection component 201: multi-source heterogeneous data system 202: user terminal device 400: first knowledge graph 401: second knowledge graph 402: entity 600: electronic device 602: processor 604: communication interface 606: memory 608: bus 610: program 301: gateway component of the control plane receives data consumption requests corresponding to N data layers, splits the data consumption requests into N sub-requests according to entity information in the data consumption requests, and distributes the sub-requests to the corresponding data layers according to the entity information in each sub-request; 302: After the N data layers receive the sub-requests respectively through the sidecar component, they query the multi-source heterogeneous data system according to the entity information in the sub-request to generate second data and return the second data to the control plane through the sidecar component; 303: After the control plane receives the second data of the N data layers through the gateway component, it fuses the second data to generate first data and provides the first data to the user through the gateway component; 501: The gateway component of the control plane receives the multi-source heterogeneous data system connection request submitted by the user; 502: The gateway component sends the multi-source heterogeneous data system connection request to the sidecar component of the corresponding data layer according to the data layer information in the multi-source heterogeneous data system connection request; 503: After the data layer receives the multi-source heterogeneous data system connection request through the sidecar component, it establishes a connection with the corresponding multi-source heterogeneous data system according to the link information. DETAILED DESCRIPTION

[0024] As mentioned earlier, intelligent business systems built with ETL tools require customized data models based on user needs. Data is retrieved from data sources using fixed data extraction rules and stored in the data warehouse. Furthermore, data association and data extraction from newly added heterogeneous systems also require customized development. Changing user needs or adding new heterogeneous systems to the intelligent business system requires redevelopment or additional custom development, which is costly and lacks applicability, making it difficult to meet the actual needs of most enterprises for big data platforms.

[0025] In addition, the source data distributed in heterogeneous systems have various business management requirements, and the business associations between data cannot be created and managed through the data model defined by the data warehouse. However, most companies in the field of intelligent manufacturing technology often need to use big data platforms to query the business associations between source data in heterogeneous systems. At the same time, they need the business associations between source data distributed in heterogeneous systems in the big data platform to be automatically updated as the business changes in real time. Obviously, the intelligent business system built with ETL tools cannot meet the actual application needs in this regard.

[0026] In an embodiment of the present application, the data federation system is divided into two parts, a control plane and a data plane, and two or more data layers are configured on the data plane. Each data layer is connected to multi-source heterogeneous data systems in different fields respectively. Each data layer is configured with a sidecar component, and the control plane is configured with a gateway component. The communication between each data layer and the control plane can be achieved through the connection between the sidecar component and the gateway component. The data plane is configured with a semantic engine, and the semantic engine can generate second data covering the business associations of its corresponding field. The control plane is configured with a federation component, and the federation component can generate first data covering the business associations of all fields by fusing the second data of each data layer. Therefore, the data federation system of the embodiment of the present application realizes the query of business associations between source data in various fields, and can adopt a loosely coupled, highly robust and low-coding architecture, with better flexibility and low development cost. It can support the flexible addition of multi-source heterogeneous data systems to meet changing application requirements.

[0027] The data union method, system, electronic device, and storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0028] Data federation system

[0029] FIG1 shows a schematic diagram of the architecture of a data federation system 100 according to an embodiment of the present application, and FIG2 shows a schematic diagram of an application scenario of the data federation system 100 according to an embodiment of the present application.

[0030] In an embodiment of the present application, the data federation system 100 employs a service mesh architecture. Referring to Figures 1 and 2 , the data federation system 100 may include a control plane 110 and a data plane 120. The data plane 120 may include two or more data layers 121, each of which is used to connect different multi-source heterogeneous data systems. Different multi-source heterogeneous data systems refer to multi-source heterogeneous data systems in different fields, which may include, but are not limited to, different businesses, different industries, and different projects. For example, assuming a data federation system is built for a region or industrial park to manage data for various industries within the region or industrial park, if the data federation system includes three data layers: data layer A, data layer B, and data layer C, it can be divided into three parts according to the international industry categories of the various industries in the region or industrial park. Data layer A can be used to connect the multi-source heterogeneous data systems of the first part of the industry, data layer B can be used to connect the multi-source heterogeneous data systems of the second part of the industry, and data layer C can be used to connect the multi-source heterogeneous data systems of the third part of the industry. Therefore, the data federation system of the embodiment of the present application can realize the joint management of multi-source heterogeneous data in various fields.

[0031] Referring to Figures 1 and 2 , the control plane 110 may include a gateway component 111 and a federation component 112. Each data layer 121 may include a sidecar component 1211 and a semantic engine 1212. Upon receiving data consumption requests corresponding to N data layers, the gateway component 111 may split the data consumption requests into N sub-requests and send them to the sidecar components 1211 corresponding to each of the N data layers. The gateway component 111 may receive the second data returned by the sidecar components 1211 of the N data layers and transmit it to the federation component 112. The federation component 112 may then provide the first data returned by the federation component 112 to the user. N is an integer greater than 1. Upon receiving the sub-requests, the sidecar component 1211 may transmit the sub-requests to the semantic engine 1212 of the current data layer and return the second data returned by the semantic engine 1212 to the gateway component 111. The semantic engine 1212 may query the multi-source heterogeneous data systems connected to the current data layer for source data based on the sub-requests to generate second data using the source data. The federation component 112 may fuse the second data from the N data layers to generate the first data.

[0032] In an embodiment of the present application, the data federation system is divided into two parts: a control plane and a data plane. The data plane is configured with two or more data layers. Each data layer is connected to multi-source heterogeneous data systems in different fields respectively. Each data layer is configured with a sidecar component 1211. The control plane is configured with a gateway component 111. Communication between each data layer and the control plane can be achieved through the connection between the sidecar component 1211 and the gateway component 111. The data plane is configured with a semantic engine 1212. The semantic engine 1212 can generate second data covering its corresponding field. The control plane is configured with a federation component 112. The federation component 112 can generate first data covering all fields by fusing the second data of each data layer. Therefore, the data federation system of the embodiment of the present application realizes the query of business associations between source data in various fields, and can adopt a loosely coupled, highly robust and low-coding architecture with better flexibility and low development cost. It can support the flexible addition of multi-source heterogeneous data systems to meet changing application requirements.

[0033] Multi-source heterogeneous data systems are used to provide multi-source data for management objects. Multi-source data includes source data from multiple businesses. Different forms of multi-source heterogeneous data systems can be deployed in different industries. In one example, a management object could be a company within a specific industry category.

[0034] Different data layers connect multi-source heterogeneous data systems in different fields, and configure different standard models at the same time. These standard models are entities. Entities can be flexibly configured in combination with the business conditions, application requirements, etc. in specific fields. In an example, assuming that data layer A connects to a multi-source heterogeneous data system in the field of intelligent manufacturing industry, multiple entities covering various specific businesses in the field of intelligent manufacturing industry can be configured in data layer A. There are many ways to configure entities. For example, the process of configuring an entity may include: creating an entity, configuring entity information for uniquely identifying the entity, and associating the entity with a standard model, etc. The embodiments of this application do not limit the configuration method of the entity.

[0035] Entity information uniquely identifies an entity. In one example, entity information may include, but is not limited to, the entity's name, number, or other identifying information. Data layer information uniquely identifies a data layer. In one example, data layer information may include, but is not limited to, the data layer's name, number, or other identifying information.

[0036] The first data may indicate the business attributes and business associations of the entity identified by the entity information in the data consumption request, and the second data may indicate the business attributes and business associations of the entity identified by the entity information in the sub-request.

[0037] The first data can be generated from a pre-built first knowledge graph, and the second data can be generated from a pre-built second knowledge graph. The first knowledge graph contains entity information for entities in all data layers of the data federation system, as well as information about the relationships between these entities. It can represent the standard models created within all data layers of the data federation system and their relationships in the form of a data view. The second knowledge graph contains entity information for entities in a data layer, as well as information about the relationships between entities in the data layer. It can represent the standard models created within a particular data layer and their relationships in the form of a data view.

[0038] The first knowledge graph can be formed by fusing the second knowledge graphs of each data layer, and the second knowledge graph can be constructed by the data layer. The semantic engines of each data layer can be used to create standard models based on international industry standards and use these standard models as entities to form a second knowledge graph. Each standard model can be used to perform predetermined algorithm processing on the source data of a specific aspect of a business to generate attribute information of the specific aspect of the business. The nodes in the second knowledge graph represent entities, that is, standard models. The connections between the nodes in the second knowledge graph can indicate the association relationship between entities, that is, the association relationship between standard models.

[0039] The gateway component 111 can be used to implement the interaction between the control plane and the data layer, and the interaction between the control plane and the user. Through the gateway component 111, the user's request can be passed to one or more data layers and the corresponding information can be returned to the user. In one example, the gateway component 111 can not only be used to receive data consumption requests or pass them to one or more data layers, but also be used to receive one or more of the following requests and distribute them to the sidecar component 1211 of the corresponding data layer: a multi-source heterogeneous data system connection request, an entity information request, and a permission configuration request. The multi-source heterogeneous data system connection request is used to add a new multi-source heterogeneous data system, the entity information request is used to update the first knowledge graph used to generate the first data, and the permission configuration request is used to assign permissions to the user.

[0040] A data consumption request may include entity information. In one possible implementation, the gateway component 111 may query a first knowledge graph pre-stored on the control plane to split the data consumption request into multiple sub-requests based on the entity information in the data consumption request and distribute them to different data layers. Each sub-request includes a piece of data layer information and a portion of the entity information in the data consumption request that corresponds to the piece of data layer information.

[0041] The first knowledge graph can indicate entities and their associations in each data layer. In one example, the first knowledge graph can be created and saved by the control plane after the data layer is registered. In one possible implementation, the joint component 112 can include a graph database 115, and the joint component 112 can be used to store the first knowledge graph through the graph database 115. In one example, the graph database 115 can also store data layer information of each data layer at the same time. After receiving the data consumption request, the gateway component 111 can query the graph database for the corresponding data layer information and complete the splitting of the data consumption request and the distribution of its sub-requests.

[0042] In one possible implementation, the fusion component 112 can fuse the second data using a first knowledge graph pre-stored in a graph database. For example, the gateway component 111 can obtain standard models from various data layers and send these standard models to the fusion component 112. The fusion component 112 then creates a first knowledge graph containing these standard models and their relationships using the graph database and stores it. The fusion component 112 can then use the first knowledge graph in the graph database to fuse the second data to obtain the first data.

[0043] In one example, the federation component 112 may also be configured to receive a second knowledge graph from a first data layer via a gateway component, and update the first knowledge graph in the graph database based on the second knowledge graph of the first data layer, so that the first knowledge graph includes entity information of entities in the first data layer, relationship information between entities in the first data layer, and relationship information between entities in the first data layer and existing entities in the first knowledge graph. Here, the first data layer refers to any registered data layer in the data federation system.

[0044] In the embodiments of the present application, the sidecar component 1211 refers to the sidecar component 1211 in the service mesh architecture. The sidecar component 1211 can establish a connection with the gateway component 111. The connection between the sidecar component 1211 and the gateway component 111 enables interaction between the data layer and the control plane. The sidecar component 1211 can be used to transfer data, messages, and / or information between the gateway component 111 and other components in the data layer. In one example, the sidecar component 1211 can also provide functions such as registration, discovery agent, flow control, and routing.

[0045] The control plane may further include a discovery registration component 113, which may be used to discover the data layer based on the data layer information so that the sidecar component 1211 of the data layer can establish a connection with the gateway component 111. In one example, the discovery registration component 113 may also be used to perform a registration operation for the data layer so that the data layer can be discovered based on the data layer information.

[0046] The control plane may also include an authentication component 114, which is used to perform user authentication so that the gateway component 111 can perform subsequent processing after the user authentication is successful. In one example, the data consumption request may include user identity information. The authentication component 114 may perform user authentication based on the user identity information in the data consumption request and generate an authentication result. The gateway component 111 may split the data consumption request after the authentication result indicates that the user authentication is successful. In one example, the multi-source heterogeneous data system connection request may also include user identity information. The authentication component 114 may perform user authentication based on the user identity information therein and generate an authentication result. The gateway component 111 may send the multi-source heterogeneous data system connection request to the corresponding data layer after the authentication result indicates that the user authentication is successful.

[0047] User identity information can indicate the user's assigned role. Different roles represent different permissions. A user can be assigned multiple roles, meaning they can have multiple user identities. In one example, the following roles can be pre-configured: data manager, developer, business partner, etc. These roles have different permissions, and the permissions for each role can be pre-defined.

[0048] The semantic engine 1212 can be used to generate second data based on the source data in the multi-source heterogeneous data system in various applicable ways. In one possible implementation, the semantic engine 1212 can also be used to create standard models for various industries based on international industry standards and save them locally in the form of a second knowledge graph. The standard model can be used to generate the business attribute information of the entity and the association relationship information between different entities using the source data and entity information. Different standard models can correspond to different entity information. After the semantic engine 1212 obtains the entity information by parsing the sub-request, it calls the standard models corresponding to these entity information in the second knowledge graph and their association relationships to process the source data from the multi-source heterogeneous data system to generate the business attribute information of the entity and the association relationship information between different entities, and uses the business attribute information and their association relationship information of the entities identified by all the entity information in the sub-request to form the second data.

[0049] Each data layer may also include a binding component 1213, which can be used to bind the multi-source heterogeneous data systems so that the semantic engine 1212 can query the multi-source heterogeneous data systems for source data. In one example, the binding component 1213 can establish and save mapping relationships between the multi-source heterogeneous data systems and entities in the data layer using a low-coding paradigm, so that the semantic engine 1212 can query the corresponding multi-source heterogeneous data systems for source data based on the entity information and generate secondary data.

[0050] Each data layer may also include a universal connection component 1214, which is used to configure various drivers and protocols so that the binding component 1213 can bind multiple heterogeneous data systems based on the drivers or protocols supported by these systems. In one example, the universal connection component 1214 may utilize an integrated data integration framework to integrate various systems in a configurable manner to support multiple drivers and protocols. In another example, the universal connection component 1214 may support new drivers by generating custom plug-ins.

[0051] In the embodiment of the present application, the data layer can adopt a lightweight semantic abstraction architecture, and the various components in the data layer can be implemented in a loosely coupled, highly robust, and low-coding manner. In one example, the data layer can also provide a data lake access interface to meet the needs of application development.

[0052] New functional components may be added to the control plane and / or data layer of the data federation system. The embodiments of the present application do not limit the specific composition of the control plane and data layer in the data federation system.

[0053] Data federation methods

[0054] FIG3 is a flow chart of a data federation method according to an embodiment of the present application. The data federation method according to the embodiment of the present application is implemented by the aforementioned data federation system. The specific implementation logic of the data federation method depends on the architecture of the data federation system and its components. As shown in FIG3 , the data federation method 300 includes the following steps:

[0055] Step 301: The gateway component of the control plane receives data consumption requests corresponding to N data layers, splits the data consumption requests into N sub-requests according to the entity information in the data consumption requests, and distributes the sub-requests to the corresponding data layers according to the entity information in each sub-request, where N is an integer greater than 1.

[0056] Each sub-request contains entity information of one data layer among all entity information in the data consumption request. In one possible implementation, the gateway component can transmit the entity information in the data consumption request to the federation component. The federation component queries the first knowledge graph stored in the graph database to determine the N data layers corresponding to the entity information in the data consumption request and transmits the data layer information of the N data layers to the gateway component. The gateway component generates N sub-requests based on the data layer information of the N data layers. Each sub-request contains a piece of data layer information and the entity information corresponding to the data layer information in the data consumption request.

[0057] Before distributing sub-requests, the gateway component can establish a connection with the sidecar component of each data layer so as to distribute the sub-requests through the connection. In one possible implementation, the discovery registration component of the control plane can discover each data layer, and the gateway component of the control plane can establish a connection with the sidecar component of each discovered data layer. In one example, in order to be able to discover the data layer, the data layer can be registered with the control plane in advance before the gateway component establishes a connection with the sidecar component. For example, the data layer can send a registration message carrying data layer information to the gateway component of the control plane through the sidecar component. The gateway component of the control plane transmits the registration request to the discovery registration component, and the discovery registration component completes the registration of the data layer.

[0058] After the data layer completes registration, the first knowledge graph in the graph database can be updated by obtaining the entity information of the data layer, so that the gateway component can split the data consumption request according to the first knowledge graph. In one possible implementation, for the registered first data layer, the control plane can send an entity information request to the sidecar component of the data layer through the gateway component. After receiving the entity information request, the sidecar component of the first data layer returns the local second knowledge graph to the gateway component of the control plane. The gateway component of the control plane receives the second knowledge graph of the first data layer and transmits it to the graph database of the control plane. The joint component of the control plane updates the first knowledge graph in the graph database and saves it, so that the first knowledge graph can include the entity information of the entities in the first data layer and the association relationship information between the entities in the first data layer, as well as the association relationship information between the entities in the first data layer and the existing entities in the first knowledge graph. Among them, the first data layer can be any data layer that has been registered with the control plane in the data federation system.

[0059] Before splitting a data consumption request, the user initiating the request can be authenticated to confirm their authorization before processing the request, ensuring the security of data assets. In one example, the data consumption request includes user identity information, which indicates the user's authorization. User authentication can be performed based on the user identity information in the data consumption request. Once authentication is successful, the data consumption request can be split by the gateway component. If authentication fails, the data consumption request splitting and subsequent processing can be omitted.

[0060] Various applicable methods can be used to assign permissions to users and configure corresponding user identity information. In one example, the control plane can receive a permission configuration request submitted by a user through a gateway component and transmit it to the authentication component of the control plane. The authentication component of the control plane generates user identity information in response to the permission configuration request and provides the user identity information to the user through the gateway component.

[0061] Step 302: After receiving the sub-requests through the sidecar component, the N data layers query the multi-source heterogeneous data system according to the entity information in the sub-request to generate second data and return the second data to the control plane through the sidecar component.

[0062] The second data is generated by the semantic engine of the data layer. For specific technical details, please refer to the record of the data federation system above and will not be repeated here.

[0063] Step 303: After receiving the second data of the N data layers through the gateway component, the control plane fuses the second data to generate first data and provides the first data to the user through the gateway component.

[0064] The first data is generated by the fusion component of the control plane. For specific technical details, please refer to the description of the data union system above, which will not be repeated here.

[0065] The second data can indicate the business attributes and business associations of the entity identified by the entity information in the sub-request, and the first data can indicate the business attributes and business associations of the entity identified by the entity information in the data consumption request. The second data can be generated based on the second knowledge graph, and the first data can be generated based on the first knowledge graph.

[0066] Figure 4 shows a schematic diagram of a first knowledge graph 400 formed by the fusion of second knowledge graphs 401 of three data layers. In Figure 4, black solid circles represent entities 402, and the connecting lines between the black solid circles represent the association relationships between entities 402. Referring to Figure 4, each second knowledge graph 401 covers the business associations of source data in different fields, and the first knowledge graph 400 formed by fusing these second knowledge graphs can cover the business associations of the entire multi-field source data. It can be seen that the embodiment of the present application can generate first data that can indicate business associations in multiple fields through data union, so that users can query business associations in multiple fields.

[0067] In one possible implementation, a user can add a new multi-source heterogeneous data system to one or more data layers by initiating a request. FIG5 shows a flow chart for adding a multi-source heterogeneous data system. As shown in FIG5 , the process 500 for adding a multi-source heterogeneous data system may include the following steps:

[0068] Step 501: The gateway component of the control plane receives a multi-source heterogeneous data system connection request submitted by a user. The multi-source heterogeneous data system connection request carries link information and data layer information.

[0069] Step 502: The gateway component sends the multi-source heterogeneous data system connection request to the sidecar component of the corresponding data layer according to the data layer information in the multi-source heterogeneous data system connection request;

[0070] Step 503: After receiving the connection request from the multi-source heterogeneous data system through the sidecar component, the data layer establishes a connection with the corresponding multi-source heterogeneous data system according to the link information.

[0071] The link information can indicate the connection address of the multi-source heterogeneous data system. The binding component of the data layer can query the driver and / or protocol of the multi-source heterogeneous data system from the driver and / or protocol configured by the universal connection component based on the link information, and use the queried driver and / or protocol to establish a connection between the multi-source heterogeneous data system and the data layer based on the link information.

[0072] electronic devices

[0073] FIG6 is a schematic diagram of an electronic device provided in Example 4 of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device. Referring to FIG6 , the electronic device 600 provided in the embodiment of the present application includes: a processor 602, a communications interface 604, a memory 606, and a bus 608. Among them:

[0074] The processor 602 , the communication interface 604 , and the memory 606 communicate with each other via a bus 608 .

[0075] The communication interface 604 is used to communicate with other electronic devices or servers.

[0076] The processor 602 is configured to execute the program 610 , and specifically may execute the relevant steps in the above-mentioned data union method embodiment.

[0077] Specifically, the program 610 may include program codes, which include computer operation instructions.

[0078] Processor 602 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0079] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0080] The program 610 may be specifically configured to enable the processor 602 to execute the data joining method in any of the aforementioned embodiments.

[0081] The specific implementation of each step in program 610 can be found in the corresponding descriptions of the corresponding steps and units in the above-mentioned data union method embodiment, and will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding process descriptions in the above-mentioned method embodiment, and will not be repeated here.

[0082] Through the electronic device of this embodiment, by dividing the data federation system into two parts, the control plane and the data plane, the data plane is configured with two or more data layers, each data layer is connected to multi-source heterogeneous data systems in different fields respectively, each data layer is configured with a sidecar component, and the control plane is configured with a gateway component. The communication between each data layer and the control plane can be realized through the connection between the sidecar component and the gateway component. The data plane is configured with a semantic engine, and the semantic engine can generate second data covering the business associations of its corresponding field. The control plane is configured with a federation component, and the federation component can generate first data covering the business associations of all fields by fusing the second data of each data layer. Therefore, the data federation system of the embodiment of the present application realizes the query of business associations between source data in various fields, and can adopt a loosely coupled, highly robust and low-coding architecture, with better flexibility and low development cost. It can support the flexible addition of multi-source heterogeneous data systems to meet changing application requirements.

[0083] Computer-readable storage medium

[0084] This application also provides a computer-readable storage medium storing instructions for causing a machine to execute the data union method described herein. Specifically, a system or device equipped with a storage medium can be provided, wherein the storage medium stores software program code that implements the functions of any of the above-described embodiments, and a computer (or CPU or MPU) of the system or device can read and execute the program code stored in the storage medium.

[0085] In this case, the program code read from the storage medium itself can realize the function of any one of the above embodiments, so the program code and the storage medium storing the program code constitute part of this application.

[0086] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD+RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer via a communications network.

[0087] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0088] In addition, it can be understood that the program code read from the storage medium is written into a memory provided in an expansion board inserted into the computer or into a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0089] Computer program product

[0090] The present application also provides a computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions. When executed, the computer-executable instructions cause at least one processor to perform the data unification methods provided in the aforementioned embodiments. It should be understood that each solution in this embodiment has the corresponding technical effects of the aforementioned method embodiments and will not be further elaborated here.

[0091] It should be noted that not all steps and modules in the above-mentioned processes and device structure diagrams are required, and certain steps or modules can be omitted according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or may be implemented by certain components in multiple independent devices.

[0092] Nouns and pronouns referring to persons in this patent application are not limited to a specific gender.

[0093] In the above embodiments, the hardware module can be implemented mechanically or electrically. For example, a hardware module can include a permanent dedicated circuit or logic (such as a dedicated processor, FPGA or ASIC) to complete the corresponding operation. The hardware module can also include programmable logic or circuits (such as a general-purpose processor or other programmable processors), which can be temporarily set by software to complete the corresponding operation. The specific implementation method (mechanical method, or dedicated permanent circuit, or temporarily set circuit) can be determined based on cost and time considerations.

[0094] The present application has been presented and described in detail above through the accompanying drawings and preferred embodiments. However, the present application is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present application, and these embodiments are also within the scope of protection of the present application.

Claims

1. A data federation system (100), comprising: A control plane and a data plane, wherein the data plane is configured with two or more data layers, each of which is used to connect different multi-source heterogeneous data systems; wherein the control plane includes a gateway component and a federation component, and each data layer includes a sidecar component and a semantic engine; The gateway component is configured to, after receiving data consumption requests corresponding to N data layers, split the data consumption requests into N sub-requests and send them to the sidecar components corresponding to the N data layers respectively, receive the second data returned by the sidecar components of the N data layers and transmit them to the joint component, and provide the first data returned by the joint component to the user; The sidecar component is configured to receive the sub-request and transmit it to the semantic engine of the current data layer, and return the second data returned by the semantic engine to the gateway component; The semantic engine is used to query the multi-source heterogeneous data system connected to the current data layer for source data according to the sub-request and generate the second data based on the source data; The combining component is used to fuse the second data of the N data layers to generate the first data; Wherein, N is an integer greater than 1.

2. The data federation system according to claim 1, wherein: The data consumption request includes entity information; each sub-request includes entity information of a data layer in the entity information of the data consumption request, the first data is used to indicate the business attributes and business associations of the entity identified by the entity information in the data consumption request, and the second data is used to indicate the business attributes and business associations of the entity identified by the entity information in the sub-request.

3. The data federation system according to claim 1, wherein: The joint component includes a graph database, which is used to store a first knowledge graph through the graph database. The first knowledge graph contains entity information in each data layer and association relationship information of entities identified by the entity information, so that the gateway component can split the data consumption request according to the first knowledge graph and determine the data layer corresponding to each sub-request.

4. The data federation system according to claim 3, wherein: The union component is further used to receive the second knowledge graph from the first data layer through the gateway component, and update the first knowledge graph in the graph database according to the second knowledge graph of the first data layer, so that the first knowledge graph can include entity information of entities in the first data layer and association relationship information between entities in the first data layer, as well as association relationship information between entities in the first data layer and existing entities in the first knowledge graph; The second knowledge graph of the first data layer includes entity information of entities in the first data layer and association relationship information between entities in the first data layer.

5. The data federation system according to claim 4, wherein: The semantic engine of each data layer is also used to create a standard model according to industry standards and form the second knowledge graph using the standard model as an entity.

6. The data federation system according to claim 3, wherein: The gateway component is also used to receive the following requests and distribute them to the sidecar components of the corresponding data layer: multi-source heterogeneous data system connection request, entity information request, and permission configuration request. The multi-source heterogeneous data system connection request is used to add a new multi-source heterogeneous data system, the entity information request is used to update the first knowledge graph, and the permission configuration request is used to assign permissions to users.

7. The data federation system according to claim 1, wherein: The control plane also includes a discovery registration component, which is used to discover the data layer according to the data layer information, so that the sidecar component of the data layer can establish a connection with the gateway component.

8. The data federation system according to claim 7, wherein: The discovery registration component is further used to perform a registration operation of the data layer so as to be able to discover the data layer according to the data layer information.

9. The data federation system according to claim 1, wherein: The control plane also includes: an authentication component for performing user authentication so that the gateway component can perform subsequent processing after the user authentication is passed.

10. The data federation system according to claim 9, wherein: The authentication component is further used to allocate the user identity information to the user in response to the permission configuration request received by the gateway component.

11. A data union method (300), the data union method being implemented by the data union system according to any one of claims 1 to 10, the method comprising: The gateway component of the control plane receives data consumption requests corresponding to N data layers, splits the data consumption requests into N sub-requests according to the entity information in the data consumption requests, and distributes the sub-requests to the corresponding data layers according to the entity information in each sub-request, wherein each sub-request contains entity information of one data layer among all entity information of the data consumption request; After receiving the sub-requests through the sidecar components, the N data layers query the multi-source heterogeneous data system according to the entity information in the sub-requests to generate second data, and return the second data to the control plane through the sidecar components, where the second data is used to indicate the business attributes and business associations of the entity identified by the entity information in the sub-requests; After the control plane receives the second data of the N data layers through the gateway component, the second data is merged to form Generate first data and provide the first data to the user through the gateway component, the first data being used to indicate the business attributes and business associations of the entity identified by the entity information in the data consumption request; Wherein, N is an integer greater than 1.

12. The method according to claim 11, wherein: The step of splitting the data consumption request into N sub-requests according to the entity information in the data consumption request includes: The gateway component transmits the entity information in the data consumption request to the joint component; The joint component queries the first knowledge graph stored in the graph database to determine N data layers corresponding to the entity information in the data consumption request and transmits the data layer information of the N data layers to the gateway component; The gateway component generates N sub-requests according to the data layer information of the N data layers, and each of the sub-requests includes one piece of the data layer information and entity information corresponding to the data layer information in the data consumption request.

13. The method according to claim 12, wherein: Before distributing the sub-requests to corresponding data layers according to the entity information in each sub-request, the method further includes: The discovery registration component of the control plane discovers each data layer; The gateway component of the control plane establishes a connection with the discovered sidecar components of each data layer, so as to send the sub-request to the sidecar component of the data layer through the connection.

14. The method according to claim 13, further comprising: The data layer sends a registration request to the gateway component of the control plane through the sidecar component, and the registration request carries the data layer information; The gateway component of the control plane transmits the registration request to the discovery registration component, and the discovery registration component completes the registration of the data layer, so that the discovery registration component of the control plane can discover the data layer.

15. The method according to any one of claims 11 to 14, further comprising: The control plane sends an entity information request to the sidecar component of the first data layer through the gateway component; After receiving the entity information request, the sidecar component of the first data layer returns the local second knowledge graph to the gateway component of the control plane; The gateway component of the control plane receives the second knowledge graph of the data layer and transmits it to the joint component of the control plane, and the joint component updates the first knowledge graph so that the first knowledge graph can include entity information of entities in the first data layer and association relationship information between entities in the first data layer, as well as information about entities in the first data layer and first There is already information about the association relationships between entities in the knowledge graph; The second knowledge graph of the first data layer includes entity information of entities in the first data layer and association relationship information between entities in the first data layer.

16. The method according to claim 11, wherein: The data consumption request includes user identity information, and the identity information is used to indicate the user's authority; Before splitting the data consumption request into sub-requests according to the entity information in the data consumption request, the method further includes: performing user authentication according to the user identity information in the data consumption request, so as to split the data consumption request through the gateway component after the authentication is passed.

17. The method according to claim 16, further comprising: The control plane receives the permission configuration request submitted by the user through the gateway component and transmits it to the authentication component of the control plane; The authentication component of the control plane generates user identity information in response to the permission configuration request, and provides the user identity information to the user through the gateway component.

18. The method according to claim 11, further comprising: The gateway component of the control plane receives a multi-source heterogeneous data system connection request submitted by a user, wherein the multi-source heterogeneous data system connection request carries link information and data layer information; The gateway component sends the multi-source heterogeneous data system connection request to the sidecar component of the corresponding data layer according to the data layer information in the multi-source heterogeneous data system connection request; After receiving the multi-source heterogeneous data system connection request through the sidecar component, the data layer establishes a connection with the corresponding multi-source heterogeneous data system according to the link information.

19. An electronic device (600), comprising: A processor (602), a communication interface (604), a memory (606) and a bus (608), wherein the processor (602), the communication interface (604) and the memory (606) communicate with each other via the bus (1108); The memory (606) is used to store at least one executable instruction, and the executable instruction enables the processor (602) to perform operations corresponding to the method as described in any one of claims 11-18.

20. A computer-readable storage medium having computer instructions stored thereon, wherein when the computer instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 11 to 18.

21. A computer program product tangibly stored on a computer readable medium and comprising computer executable instructions which, when executed, cause at least one processor to perform the method according to any one of claims 11-18.

Images