A method for solving cross-domain heterogeneous data joint retrieval
By building an engine management platform and a distributed data collaboration engine, the problems of industry data resources are solved and the problems of industry data resources are realized, the integration of cross-departmental data sources is realized, business management and defense capabilities are improved, and data value is maximized. use.
Patent Information
- Application Number
- CN202111072328.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-14
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-09-14
AI Technical Summary
The industry's data resources are scattered and departmental barriers have led to insufficient data integration, insufficient coverage, different standards, low efficiency, lack of effective data processing methods, difficult to meet the needs of information sharing, and insufficient data value can be fully explored.
Build an engine management platform, structure a distributed data collaboration engine, carry out data access engine development, build a multi-domain data source management platform, build a collaborative data cache library, open a collaborative service bus, and finally output a complete collaborative computing bus platform, using Apache Ambari The automated deployment and tuning of Presto components realizes a multi-node distributed architecture and supports the fusion computing and analysis of heterogeneous data sources.
It realizes heterogeneous data source fusion computing across business departments, forms an overall solution, meets cross-department collaborative computing of big data, improves business management and defense capabilities, breaks data barriers, and maximizes the utilization of data value.
Smart Images

Figure CN113886457B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data, and specifically provides a method for solving cross-domain heterogeneous data joint retrieval. Background Art
[0002] The types of industry data are complex and the sources are diverse, which makes it difficult to collect data sources. All departments and cities have certain data resources, but departmental barriers and organizational barriers have not been completely broken down, and the data resources that should be gathered have not been gathered. The "data island" phenomenon has existed for a long time, resulting in few sources of big data integration, narrow coverage, inconsistent standards, insufficient efforts and low efficiency.
[0003] The problems of data surplus and data hunger coexist, with data surplus and other data hunger; the superior authorities have data surplus and the grassroots departments are data hungry. At the same time, the data management system is not sound, data management is out of touch with business needs, some mechanisms and regulations are not conducive to the in-depth development of data sharing, and it is difficult to meet the urgent needs of information sharing. The data environment of each department presents a chimney construction model, resulting in completely independent and isolated departmental data. There is a lack of effective data processing methods, and the data value mining is insufficient, which cannot ensure that the maximum value of data serves actual business needs.
[0004] In view of the current business situation of difficult data sharing faced by the industry, there is an urgent need to build an integrated data collaborative computing bus to connect the networks of various business departments while ensuring that the existing big data centers of each business department are not affected. Summary of the invention
[0005] The present invention aims at the above-mentioned deficiencies of the prior art and provides a method for solving the cross-domain heterogeneous data joint retrieval with strong practicality.
[0006] The technical solution adopted by the present invention to solve its technical problem is:
[0007] A method for solving cross-domain heterogeneous data joint retrieval has the following steps:
[0008] S1. Build an engine management platform;
[0009] S2, architecture distributed data collaboration engine;
[0010] S3, data access engine;
[0011] S4, multi-domain data source management platform;
[0012] S5. Build a collaborative data cache library;
[0013] S6, open collaborative service bus;
[0014] S7. Output a complete collaborative computing bus platform.
[0015] Furthermore, the Apache open source product Ambari is used to package the collaborative engine component presto into components. Ambari performs automatic deployment, management and monitoring of presto, and orchestrates the presto component in a multi-node distributed architecture.
[0016] Furthermore, when the Presto component performs multi-node distributed architecture orchestration, the Presto coordinator is installed on one of the nodes, and the Presto workers are installed on the remaining nodes, forming a multi-node distributed architecture mode.
[0017] Furthermore, in step S2, a database is developed based on the collaborative computing engine Presto, and Presto's plugin is used for integration, and the developed database is uniformly converted into SQL syntax that complies with the SQL92 standard through the internal logic of the connector;
[0018] At the same time, according to the actual resource situation of the collaborative computing cluster, the number of threads, concurrency and JVM parameters of Presto are tuned and configured, and then the Presto components are automatically deployed through the Ambari management platform to realize the deployment of the distributed collaborative computing engine.
[0019] Further, in step S3, based on the collaborative computing bus engine Presto, a data access engine with an analyzer, an optimizer, a decomposer and a scheduler is developed, and the data access engine complies with the SQL92 standard for SQL input and parsing;
[0020] It is adapted and connected with the Presto access engine to provide input, parsing, execution, and scheduling capabilities for standard SQL.
[0021] Furthermore, in step S4, based on the collaborative computing engine Presto, a multi-domain data source management platform is implemented, mainly for managers and developers. The manager uses the task monitoring screen based on the Presto engine to approve or reject the developer's data source permission opening and API access application, and at the same time, configure the access operation permissions of various data source library tables and columns for various users.
[0022] Furthermore, in step S5, based on the automatic deployment capability of the engine management platform, two types of cache libraries, Hive and MySQL, are constructed, and the results after preliminary calculation are stored in the two types of cache libraries, Hive and MySQL. If the results after preliminary calculation are not comprehensive, the developer performs secondary parallel calculation of the data.
[0023] Furthermore, in step S6, the data sources and API services registered and published by various departments are opened to upper-level applications through the collaborative service bus. Developers can browse various heterogeneous data sources and API detailed information published by various departments through the collaborative service bus and apply for online approval.
[0024] Compared with the prior art, the method for solving cross-domain heterogeneous data joint retrieval of the present invention has the following outstanding beneficial effects:
[0025] The present invention can solve the problem of integrated computing and analysis of heterogeneous data sources across business departments, form an overall solution for joint analysis of heterogeneous data sources, and realize the integration and docking capabilities of heterogeneous data sources (relational databases, MPP databases, Hadoop big databases, PostgreSQL databases, NoSQL databases, etc.) of departments such as science and technology and information technology, meet the needs of cross-departmental big data collaborative computing, integrate science and technology and information technology data to provide comprehensive business analysis services in the form of services to the outside world, and improve business management and defense capabilities. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0027] Figure 1 It is an architectural diagram of a method for solving cross-domain heterogeneous data joint retrieval (I);
[0028] Figure 2 It is an architectural diagram of a method for solving cross-domain heterogeneous data joint retrieval (I);
[0029] Figure 3 It is a schematic diagram of the structure of building an engine management platform in a method for solving cross-domain heterogeneous data joint retrieval;
[0030] Figure 4 This is an architecture diagram of a distributed data collaboration engine in a method for solving cross-domain heterogeneous data joint retrieval;
[0031] Figure 5 This is a diagram of the distributed deployment architecture of a data collaboration engine in a method for solving cross-domain heterogeneous data joint retrieval;
[0032] Figure 6 It is a data access engine in the method of solving cross-domain heterogeneous data joint retrieval;
[0033] Figure 7It is an architectural diagram of a multi-domain data source management platform in a method for solving cross-domain heterogeneous data joint retrieval;
[0034] Figure 8 It is an architectural diagram of building a collaborative data cache library in a method for solving cross-domain heterogeneous data joint retrieval;
[0035] Fig. 9 The diagram is an architectural diagram of an open collaborative service bus in a method for solving cross-domain heterogeneous data joint retrieval. DETAILED DESCRIPTION
[0036] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] A best embodiment is given below:
[0038] like Figure 1-2 As shown, a method for solving cross-domain heterogeneous data joint retrieval in this embodiment includes two parts: a data access engine and a data collaboration engine. The data access engine mainly provides a SQL92 standard JDBC access engine for upper-level applications. The data collaboration engine provides joint query, analysis and calculation of multi-source heterogeneous data sources, and also provides two types of collaborative data caches, Hive and MySQL, for temporary storage of process data. Then, through the "multi-domain data source management" system, various data sources in the W domain, J domain, and K domain are connected to the collaborative computing bus through various database connector plugins. The engine management platform system mainly provides functions such as automated deployment, online monitoring and operation of the collaborative computing bus; finally, the registered data source and webservice interface are provided to various business departments through the collaborative service bus.
[0039] The specific technical implementation process is as follows:
[0040] S1. Build an engine management platform:
[0041] like Figure 3As shown, the Apache open source product Ambari is used. Ambari is a Web-based distributed architecture management tool for creating, managing and monitoring Hadoop clusters and collecting a large amount of information from cluster nodes and services. With the open interface of Ambari, the collaborative engine component presto is packaged in components, and the automatic deployment, management and monitoring of presto based on Ambari are realized. The presto component is arranged in a multi-node distributed architecture. Presto coordinator is installed on one node, and Presto workers are installed on other nodes, so that it forms a multi-node distributed architecture model with high-performance data parallel computing capabilities.
[0042] S2. Architecturally distributed data collaboration engine:
[0043] like Figure 4 , 5 As shown in the figure, based on the collaborative computing engine Presto, various database connectors are developed, such as Oracle cnonector, HBase connector, MPP connector, etc., and integrated as Presto plugins. It is necessary to convert various types of database structures into SQL syntax that conforms to the SQL92 standard through the internal logic of the connector. At the same time, according to the actual resource situation of the collaborative computing cluster, the number of threads, concurrency, JVM and other parameters of Presto are tuned and configured.
[0044] At the same time, the Ambari management platform is used to automatically deploy Presto components and implement the deployment of a distributed collaborative computing engine.
[0045] S3, data access engine:
[0046] like Figure 6 As shown, based on the collaborative computing bus engine Presto, taking into account the heterogeneity of the access data source, a data access engine with the capabilities of analyzer (Parser), optimizer (Optimizer), decomposer (Fragmenter) and scheduler (Scheduler) is developed, and SQL input and parsing that conform to the SQL92 standard are adapted and connected with the access engine of Presto, thereby meeting the input, parsing, execution and scheduling capabilities of standard SQL.
[0047] S4, Multi-domain data source management platform:
[0048] like Figure 7As shown in the figure, based on the collaborative computing engine Presto, a multi-domain data source management platform is implemented, which mainly provides console functions for two types of role users: managers and developers. A task monitoring screen based on the Presto engine is provided for managers, which supports managers to register, access, monitor and manage various heterogeneous data sources and API interfaces online, and approve or reject developers' data source permission opening and API access applications. At the same time, it can configure access operation permissions for various data source library tables and columns for various users; it provides developers with online viewing and application functions for data sources, and monitors and manages the opened data sources and APIs.
[0049] S5. Build a collaborative data cache library:
[0050] like Figure 8 As shown in the figure, after each department uses the data collaboration engine to perform memory-based distributed parallel computing on heterogeneous data sources, the final result data may not be formed, and it is often necessary to connect to other departments' data sources for secondary computing. At this time, it is necessary to build a collaborative data cache library in the collaborative computing engine to provide the results of the preliminary calculations and temporarily store them in the data cache library to facilitate developers to perform secondary parallel computing of the data. Based on the automatic deployment capabilities of the engine management platform, two types of cache libraries, Hive and MySQL, can be built.
[0051] S6. Open collaborative service bus:
[0052] like Fig. 9 As shown, finally, the data sources and API services registered and published by various departments are opened to upper-level applications through the collaborative service bus. Developers can browse various heterogeneous data sources and API details published by various departments through the collaborative service bus, and can apply for approval online, thereby ensuring that each department can access the required resources based on their actual business needs.
[0053] S7. Output a complete collaborative computing bus platform:
[0054] At this point, after the above steps, a holistic solution to the joint retrieval of heterogeneous data has been constructed, which can effectively break down the data barriers between departments, ensure the maximization of data utilization value, realize data sharing among business departments, and effectively improve the management and defense capabilities of business departments.
[0055] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any appropriate changes or substitutions made by any ordinary technician in the technical field that conform to the claims of the present invention for solving cross-domain heterogeneous data joint retrieval shall fall within the patent protection scope of the present invention.
[0056] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for solving cross-domain heterogeneous data joint retrieval, characterized in that: The following steps are involved: S1. Build an engine management platform; The Apache open source product Ambari is used to package the collaborative engine component presto into components. Ambari performs automatic deployment, management, and monitoring of presto, and orchestrates the presto component into a multi-node distributed architecture. When the presto component performs multi-node distributed architecture orchestration, Prestocoordinator is installed on one of the nodes, and Presto workers are installed on the remaining nodes, forming a multi-node distributed architecture mode; S2, architecture distributed data collaboration engine; Develop a database based on the collaborative computing engine Presto, use the Presto plugin for integration, and convert the developed database into SQL syntax that complies with the SQL92 standard through the connector's internal logic; At the same time, according to the actual resource situation of the collaborative computing cluster, the number of threads, concurrency and JVM parameters of Presto are optimized and configured, and then the Presto components are automatically deployed through the Ambari management platform to realize the deployment of the distributed collaborative computing engine; S3, data access engine; Based on the collaborative computing bus engine Presto, develop a data access engine with analyzer, optimizer, decomposer and scheduler. The data access engine complies with the SQL92 standard for SQL input and parsing. Through adaptation and docking with Presto's access engine, it is used for input, parsing, execution and scheduling capabilities of standard SQL; S4, multi-domain data source management platform; Based on the collaborative computing engine Presto, a multi-domain data source management platform is implemented. For managers and developers, the manager has a large screen for monitoring tasks based on the Presto engine; the manager can approve or reject the developer's data source permission opening and API access application, and configure the access operation permissions of various data source library tables and columns for various users; S5. Build a collaborative data cache library; Based on the automatic deployment capability of the engine management platform, two types of cache libraries, Hive and MySQL, are built to store the results of preliminary calculations in the two types of cache libraries. If the results of preliminary calculations are not comprehensive, developers perform secondary parallel calculations on the data. S6, open collaborative service bus; The data sources and API services registered and published by various departments are opened to upper-level applications through the collaborative service bus. Developers can browse various heterogeneous data sources and API details published by various departments through the collaborative service bus and conduct online application approval. S7. Output a complete collaborative computing bus platform.
Citation Information
Cited By
Cross-source data collaborative analysis method and system based on AI model
CN119784415A
