Data sharing method and apparatus

Abstract

The application discloses a data sharing method and device, and relates to the technical field of big data operation. A specific implementation of the method comprises the following steps: receiving a data sharing instruction for a target database and a target cluster; determining a target external table corresponding to the data sharing instruction, wherein the target external table is stored on the target cluster; acquiring a data sharing protocol of the target database and the target cluster; and based on the data sharing protocol, completing relevant processing of the data sharing instruction by using the target external table. The implementation can simply and efficiently realize data sharing between the target database and the target cluster.

Description

Technical Field

[0001] This invention relates to the field of big data operation technology, and in particular to a data sharing method and apparatus. Background Technology

[0002] With the rapid development of Internet technology, the application data of various industries is growing at an exponential rate. The demand for data acquisition, data processing, cross-analysis and deep mining is also increasing. A single data storage and analysis platform can no longer meet the analysis or development needs of various applications. Applications have begun to choose appropriate database products and analysis platform tools according to their own development needs, data characteristics and data volume. The database processing field has also shifted from traditional single-node databases to distributed databases.

[0003] MPP databases and Hadoop clusters are widely used in big data storage and computing. HDFS, due to its high cost-effectiveness, is often used in enterprise data archiving systems and is also suitable for scenarios such as data cleaning and organization. MPP databases are widely used in data warehouses, storing large amounts of core data while supporting daily analysis and business decisions. In current technologies, achieving data sharing between MPP databases and Hadoop clusters requires configuring a Hadoop client specifically on the MPP database nodes, but the client configuration process is cumbersome and prone to errors. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a data sharing method and apparatus that can simply and efficiently realize data sharing between a target database and a target cluster.

[0005] In a first aspect, embodiments of the present invention provide a data sharing method, including:

[0006] Receive data sharing instructions for the target database and target cluster;

[0007] Determine the target external table corresponding to the data sharing instruction, and the target external table is stored on the target cluster;

[0008] Obtain the data sharing protocol between the target database and the target cluster;

[0009] Based on the data sharing protocol, the relevant processing of the data sharing instructions is completed using the target external table.

[0010] Optionally, the data sharing instruction is used to share data from the target database to the target cluster;

[0011] Determining the target external table corresponding to the data sharing instruction includes:

[0012] Based on the data sharing instruction, the storage path of the target external table is determined, and the storage path is located on the target cluster;

[0013] Based on the data sharing protocol, the target external table is created in the storage path;

[0014] The process of processing the data sharing instruction based on the data sharing protocol and utilizing the target external table includes:

[0015] Based on the data sharing protocol, the shared data is written into the target external table using the target database to complete the sharing of data from the target database to the target cluster.

[0016] Optionally, the data sharing instruction is used to share data from the target cluster to the target database;

[0017] The process of processing the data sharing instruction based on the data sharing protocol and utilizing the target external table includes:

[0018] Based on the data sharing protocol, data is read from the target external table using the target database to complete the sharing of data from the target cluster to the target database.

[0019] Optionally, before obtaining the data sharing protocol between the target database and the target cluster, the method further includes:

[0020] The client installation file for the target cluster is saved on the target node of the target database;

[0021] Based on the access information of the target file system of the target cluster, create the configuration file of the target cluster on the target node;

[0022] According to the program in the client installation file, a data sharing protocol is created between the database and the target cluster.

[0023] Optionally, the target database is an MPP database, the target cluster is a Hadoop cluster, and the target file system is a distributed file system HDFS.

[0024] Optionally, saving the client installation file of the target cluster on the target node of the target database includes:

[0025] Call the Hadoop cluster client development interface to generate the client installation file;

[0026] Save the client installation file to the target node of the target database.

[0027] Optionally, the step of creating a data sharing protocol between the database and the target cluster based on the program in the client installation file includes:

[0028] The program in the client installation file is invoked to create an external table read function on the target node;

[0029] The program in the client installation file is invoked to create an external table write function on the target node;

[0030] The data sharing protocol is generated based on the read function and the write function;

[0031] Grant the execution rights of the data sharing protocol to database users with preset permissions.

[0032] Optionally, the step of processing the data sharing instruction based on the data sharing protocol and utilizing the target external table includes:

[0033] Based on the data sharing protocol and the configuration file, the relevant processing of the data sharing instructions is completed using the target external table.

[0034] Optionally, after receiving the data sharing instruction for the target database and the target cluster, the method further includes:

[0035] Obtain the external protocol of the target database;

[0036] Based on the external protocol, data in external storage is imported into the target database, and / or data in the target database is imported into external storage.

[0037] Optionally, before obtaining the external protocol of the target database, the method further includes:

[0038] Receive configuration instructions for the external protocol, the configuration instructions including: protocol type and data format;

[0039] The external protocol is generated based on the protocol type and the data format.

[0040] Secondly, embodiments of the present invention provide a data sharing device, comprising:

[0041] The instruction receiving module is used to receive data sharing instructions for the target database and the target cluster;

[0042] The table determination module is used to determine the target external table corresponding to the data sharing instruction, and the target external table is stored on the target cluster.

[0043] The protocol acquisition module is used to acquire the data sharing protocol between the target database and the target cluster;

[0044] The processing module is used to complete the relevant processing of the data sharing instructions based on the data sharing protocol and the target external table.

[0045] Optionally, the data sharing instruction is used to share data from the target database to the target cluster;

[0046] The table determination module is specifically used for:

[0047] Based on the data sharing instruction, the storage path of the target external table is determined, and the storage path is located on the target cluster;

[0048] Based on the data sharing protocol, the target external table is created in the storage path;

[0049] The processing module is specifically used for:

[0050] Based on the data sharing protocol, the shared data is written into the target external table using the target database to complete the sharing of data from the target database to the target cluster.

[0051] Thirdly, embodiments of the present invention provide an electronic device, including:

[0052] One or more processors;

[0053] Storage device for storing one or more programs.

[0054] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the above embodiments.

[0055] Fourthly, embodiments of the present invention provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0056] Fifthly, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the above embodiments.

[0057] One embodiment of the above invention has the following advantages or beneficial effects: the target database is a structured database, such as an MPP database. The target cluster is a cluster using a distributed file system, such as a Hadoop cluster. Through a data sharing protocol, the target database can read and write external tables stored in the target cluster, realizing data sharing between the target database and the target cluster, without any intermediate nodes or systems for data transfer during the data sharing process. Therefore, the solution of this embodiment can simply and efficiently realize data sharing between the target database and the target cluster.

[0058] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0059] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0060] Figure 1 This is a flowchart illustrating a data sharing method provided in the first embodiment of the present invention;

[0061] Figure 2 This is a flowchart illustrating a data sharing method provided in the second embodiment of the present invention;

[0062] Figure 3 This is a flowchart illustrating a data sharing method provided in the third embodiment of the present invention;

[0063] Figure 4 This is a schematic diagram of a process for sharing an MPP database with a Hadoop cluster, provided in the fourth embodiment of the present invention;

[0064] Figure 5 This is a schematic diagram of data sharing between an MPP database and a Hadoop cluster provided in the fourth embodiment of the present invention;

[0065] Figure 6 This is a schematic diagram of the structure of a data sharing device provided in an embodiment of the present invention;

[0066] Figure 7 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0067] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0068] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0069] Figure 1 This is a flowchart illustrating a data sharing method provided in the first embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0070] Step 101: Receive data sharing instructions for the target database and the target cluster.

[0071] The target database can be a structured database, such as an MPP database or an Oracle database. The target cluster is a cluster using a distributed file system, such as a Hadoop cluster or a FastDFS cluster.

[0072] Data sharing instructions are used to share data between a target database and a target cluster. Data sharing instructions can be used to share data from the target database to the target cluster. Data sharing instructions can also be used to share data from the target cluster to the target database.

[0073] Step 102: Determine the target external table corresponding to the data sharing instruction, and the target external table is stored on the target cluster.

[0074] The target external table is an external table of the target database. The target database only contains the definition of the target external table, but not the data itself. The data files for the target external table are stored in the target cluster, outside the target database. When reading data, the target database loads the data from the data files in the target cluster.

[0075] Step 103: Obtain the data sharing protocol between the target database and the target cluster.

[0076] Data sharing protocols are used to facilitate data sharing between the target database and the target cluster. These protocols may include: data transmission protocols, external table read functions, external table write functions, target cluster access information, and permission information. Data transmission protocols may include: Gpfdist protocol, HTTP protocol, etc.

[0077] Step 104: Based on the data sharing protocol, use the target external table to complete the relevant processing of the data sharing instruction.

[0078] When the data sharing instruction is used to share data from the target database to the target cluster, the storage path of the target external table can be determined according to the data sharing instruction, and the storage path is located on the target cluster; based on the data sharing protocol, the target external table is created in the storage path; based on the data sharing protocol, the shared data is written into the target external table using the target database to complete the sharing of data from the target database to the target cluster.

[0079] When the data sharing instruction is used to share data from the target cluster to the target database, data can be read from the target external table using the target database based on the data sharing protocol to complete the data sharing from the target cluster to the target database.

[0080] It should be noted that different target clusters may correspond to different external tables and data sharing protocols. Using the method of this invention, the target database can perform read and write operations on different target external tables to achieve data sharing with different target clusters.

[0081] In this embodiment of the invention, the target database is a structured database, such as an MPP database. The target cluster is a cluster using a distributed file system, such as a Hadoop cluster. Through a data sharing protocol, the target database can read and write external tables stored in the target cluster, realizing data sharing between the target database and the target cluster. No intermediate nodes or systems are used for data transfer during the data sharing process. Therefore, the solution of this embodiment of the invention can simply and efficiently achieve data sharing between the target database and the target cluster.

[0082] Figure 2 This is a flowchart illustrating a data sharing method provided in the second embodiment of the present invention, as shown below. Figure 2 As shown, the method includes:

[0083] Step 201: Save the client installation file for the target cluster on the target node of the target database.

[0084] By saving the target cluster's client installation file, the original target cluster client installation is replaced. The target cluster's client installation file may include: data transfer functions, external table read functions, external table write functions, target cluster access information, permission information, etc., so that the data sharing protocol can achieve data sharing between the target database and the target cluster by encapsulating or calling the target cluster's client installation file.

[0085] For example, if the target database is an MPP database, the target cluster is a Hadoop cluster, and the target file system is a distributed file system HDFS, the client development interface of the Hadoop cluster can be called to generate the client installation file; the client installation file can then be saved to the target node of the target database.

[0086] Specifically, the native C++ libhdfs3 is used as the client to access HDFS. The C interface is implemented according to the predetermined API pattern, compiled into a .so package, and then the corresponding .so package is copied to the MPP installation directory, thus avoiding the original complex Hadoop client installation and configuration.

[0087] Step 202: Based on the access information of the target file system of the target cluster, create the configuration file of the target cluster on the target node.

[0088] The target file system can be a distributed file system, such as GFS, HDFS, or Lustre. The configuration file can include access information for the target file system of the target cluster, allowing the target database to access external tables stored in the target file system via the configuration file. The configuration file can include the following information: cluster name, host information for the target file system, port information for the target file system, authentication mode, etc.

[0089] Based on the data sharing protocol and configuration file, the target external table is used to complete the relevant processing of the data sharing instructions. Specifically, the data sharing protocol is used to determine the data transmission and sharing method between the target database and the target cluster. The configuration file is used to determine the access information of the target file system of the target cluster. Based on the data sharing protocol and configuration file, the target database can directly read and write the target external table for data sharing without the need for data transfer through other systems, saving resources and increasing efficiency.

[0090] Step 203: Create a data sharing protocol between the database and the target cluster according to the program in the client installation file.

[0091] The client installation file for the target cluster may include functions or information related to data transfer and sharing. The data sharing protocol, by encapsulating or calling the client installation file of the target cluster, enables data sharing between the target database and the target cluster.

[0092] In one embodiment of the present invention, a data sharing protocol between the database and the target cluster can be created by: calling a program in the client installation file to create an external table read function on the target node; calling a program in the client installation file to create an external table write function on the target node; generating the data sharing protocol based on the read function and the write function; and granting execution permission of the data sharing protocol to a database user with preset permissions.

[0093] The data sharing protocol encapsulates calls to external table read functions, enabling the target database to directly read data from the target external table. Similarly, it encapsulates calls to external table write functions, allowing the target database to directly write to the target external table. Therefore, based on this data sharing protocol, the target database can directly read and write to the target external table for data sharing without needing data intermediaries from other systems, saving resources and increasing efficiency.

[0094] Step 204: Receive data sharing instructions for the target database and the target cluster.

[0095] Step 205: Determine the target external table corresponding to the data sharing instruction, and the target external table is stored on the target cluster.

[0096] Step 206: Obtain the data sharing protocol between the target database and the target cluster.

[0097] Step 207: Based on the data sharing protocol, use the target external table to complete the relevant processing of the data sharing instruction.

[0098] In this embodiment of the invention, the client installation file for the target cluster is saved, replacing the original client installation for the target cluster. This eliminates the need for periodic authentication of the target cluster, enabling file access from the target database to the target file system. The configuration process is simple and efficient, and the load on the target database is reduced.

[0099] Figure 3 This is a flowchart illustrating a data sharing method provided in the third embodiment of the present invention, as shown below. Figure 3 As shown, the method includes:

[0100] Step 301: Obtain the external protocol of the target database.

[0101] External protocols are used for sharing data stored externally between target databases. External protocols may include: protocol type and data format. Protocol types may include: gpfdist, http, etc. Data formats are used to import data from the target database into files with a specified data format on external storage, or to import data from files with a specified data format on external storage into the target database. Data formats can be configured according to user needs. Data formats may include: TEXT, CSV, XLS, etc.

[0102] Before acquiring the external protocol of the target database, the process may further include: receiving configuration instructions for the external protocol, the configuration instructions including: protocol type and data format; and generating the external protocol based on the protocol type and data format. Users can configure the external protocol according to their needs to achieve data sharing between external storage and the target database.

[0103] Step 302: Based on the external protocol, import the data in the external storage into the target database, and / or import the data in the target database into the external storage.

[0104] Based on an external protocol, data from external storage can be imported or exported to the target database, enabling data sharing between the target database and external storage. External storage can be storage within the user's system or other storage specified by the user.

[0105] Step 303: Receive data sharing instructions for the target database and the target cluster.

[0106] Step 304: Determine the target external table corresponding to the data sharing instruction, and the target external table is stored on the target cluster.

[0107] Step 305: Obtain the data sharing protocol between the target database and the target cluster.

[0108] Step 306: Based on the data sharing protocol, use the target external table to complete the relevant processing of the data sharing instruction.

[0109] In this embodiment of the invention, data sharing between the target database and external storage is achieved through an external protocol. Data sharing between the target database and the target cluster is also achieved through a data sharing protocol. The solution provided by this embodiment of the invention can meet the needs of data sharing, analysis, and development among multiple systems and applications, simplifying user operations and improving the user experience in data analysis and development projects.

[0110] To make the solutions in this embodiment of the invention easier to understand, the target database is an MPP database and the target cluster is a Hadoop cluster. The data sharing scheme between MPP data and Hadoop cluster will be used as an example for explanation. Figure 4 This is a schematic diagram of an MPP and Hadoop data sharing process provided in the fourth embodiment of the present invention. Figure 4 As shown, firstly, libhdfs3 is used as the Hadoop client installed on the MPP cluster; then, a data source configuration file is introduced to centralize the configuration information of one or more Hadoop clusters; next, an external table protocol based on the HDFS file system is created, enabling MPP database users to access tables in the MPP database and structured data files in HDFS by creating readable or writable external tables; finally, users can achieve data sharing between the MPP cluster and the Hadoop cluster by creating corresponding external tables.

[0111] To simplify configuration and improve data exchange efficiency between the two systems, a data sharing scheme was proposed and implemented: by adding an external table protocol to the MPP cluster to access HDFS, each compute node on the MPP can directly connect to the Hadoop cluster without any intermediate nodes or systems for data transfer. Readable or writable external tables can be created directly in both databases to access data in the other's database. The specific implementation methods for each step are described below:

[0112] Step S01: Traditional MPP databases require configuring Hadoop clients and Java environment variables on each node in the cluster during deployment and use. They also require periodic authentication with the Hadoop cluster and can only achieve data sharing with a single HDFS system. This invention uses the native C++ libhdfs3 as the HDFS client, implementing several C interfaces according to a predetermined API pattern, compiling them into .so packages, and then copying the corresponding .so packages to the MPP installation directory, thus avoiding the previously complex Hadoop client installation and configuration.

[0113] Step S02: Configuring the gphdfs file involves creating a data source configuration file, gphdfs.conf, which centralizes access information from multiple HDFS file systems. Each HDFS is mapped using an alias, and access information for that system is configured under the corresponding alias, simplifying access configuration management. All parameters in the gphdfs.conf configuration template can be obtained from the Hadoop cluster's value configuration file, hdfs-site.xml.

[0114] The configuration file allows setting various parameters, including:

[0115] pac_cluster1: Abbreviation for Hadoop cluster 1 (users can define their own name; different names can be defined for multiple clusters).

[0116] hdfs_namenode_host: Configures the host information for HDFS;

[0117] hdfs_namenode_port: Configures the port information for HDFS;

[0118] `hdfs_auth_method`: Configures the HDFS authentication mode. Standard HDFS uses `simple`. HDFS with Kerberos uses Kerberos.

[0119] krb_principal: A user in Kerberos;

[0120] krb_principal_keytab: The location where the user-generated keytab is placed;

[0121] Hadoop_rpc_protection: Same as the configuration in hdfs core-site.xml;

[0122] is_ha_supported: Set to true if the cluster is configured for HA;

[0123] dfs.nameservices: Hadoop cluster name;

[0124] dfs.ha.namenodes.mycluster: Hostname or IP information of the Hadoop cluster namenode.

[0125] Step S03: After configuring the HDFS client and cluster parameters, the corresponding external protocol for the HDFS file system can be created in the MPP database. This allows MPP database tables to be written to or read from files in HDFS. The MPP database supports multiple external table protocols such as gpfdist and HTTP. When the built-in protocols cannot meet the needs, custom protocols are also supported. Users can define their own protocols to access their external data. The data format can be standard TEXT or CSV, or a custom format. Through external table protocols, users can directly import and export data between their system and the MPP database.

[0126] In the HDFS client installation in step S01, the compiled .so file has been placed in the installation directory. Here, the interface in the database function call .so file is encapsulated, and then the encapsulated function is given an external table protocol. The usage rights of the protocol are then granted to the database user.

[0127] The specific implementation steps may include: creating external table read functions and external table write functions; encapsulating the read and write functions into the gphdfs protocol; and finally, granting the gphdfs protocol to all database users.

[0128] Step S04: When creating a readable or writable external table in MPP data, by defining the external table file pointing to the path location in the HDFS system, data in a file block at a specified location can be read or data can be written to a specified location. The steps for an MPP cluster to unload data to a Hadoop cluster are: create a writable external table, initialize the external data connection, insert data into the writable external table, send data to the external system, and close the external connection to complete the data unloading. The steps for an MPP cluster to load data from a Hadoop cluster are: create a readable external table, initialize the external connection, query the readable external table to receive data, and close the external connection. The SQL code for these two data exchange methods in the database can be simplified as follows:

[0129] The data offloading example includes the following steps: (1) Create a writable external table pointing to the / tmp directory of the h cluster.

[0130] (2) In HDFS systems where data can be transmitted simply by inserting data.

[0131] The data loading instance includes the following steps: (1) Create a readable external table.

[0132] (2) Read the data used.

[0133] Figure 5 This is a schematic diagram illustrating data sharing between an MPP database and a Hadoop cluster, provided in the fourth embodiment of the present invention. Figure 5 As shown, after the data in MPP is unloaded into HDFS, the Hadoop component Hive is used for data reading, writing and analysis. Hive is a data warehouse tool based on Hadoop that can map structured data files to a database table and provide complete SQL query functions, thereby realizing data sharing between the two types of databases.

[0134] This invention's solution replaces Hadoop client installation by distributing the included libhdfs3.so package to all nodes in the MPP cluster. This eliminates the need for periodic authentication with the Hadoop cluster, enabling file access from the MPP database to the HDFS system. The configuration process is simple and efficient, reducing the load on the MPP cluster. Data sharing between the MPP cluster and multiple HDFS file systems can be achieved by adding Hadoop cluster information to the gphdfs file. Data sharing is directly achieved using external table read / write, eliminating the need for data transfer through other systems, saving resources and increasing efficiency. Through external protocol definitions, various data file formats, such as CSV, TXT, and DAT, can be read and written.

[0135] This invention implements data sharing between different types of databases through a custom database sharing protocol. The key technical points and protections lie in the configuration steps for data sharing, the definition of the external table protocol, and the method for creating external tables. Furthermore, this invention utilizes configuration files to achieve data sharing between an MPP cluster and multiple Hadoop clusters, optimizing the user experience of database applications and simplifying the data sharing process.

[0136] Figure 6 This is a schematic diagram of the structure of a data sharing device provided in one embodiment of the present invention, as shown below. Figure 6 As shown, the device includes:

[0137] Instruction receiving module 601 is used to receive data sharing instructions for the target database and the target cluster;

[0138] Table determination module 602 is used to determine the target external table corresponding to the data sharing instruction, wherein the target external table is stored on the target cluster;

[0139] Protocol acquisition module 603 is used to acquire the data sharing protocol between the target database and the target cluster;

[0140] The processing module 604 is used to complete the relevant processing of the data sharing instruction based on the data sharing protocol and using the target external table.

[0141] Optionally, the data sharing instruction is used to share data from the target database to the target cluster;

[0142] The table determination module 602 is specifically used for:

[0143] Based on the data sharing instruction, the storage path of the target external table is determined, and the storage path is located on the target cluster;

[0144] Based on the data sharing protocol, the target external table is created in the storage path;

[0145] The processing module 604 is specifically used for:

[0146] Based on the data sharing protocol, the shared data is written into the target external table using the target database to complete the sharing of data from the target database to the target cluster.

[0147] Optionally, the data sharing instruction is used to share data from the target cluster to the target database;

[0148] The processing module 604 is specifically used for:

[0149] Based on the data sharing protocol, data is read from the target external table using the target database to complete the sharing of data from the target cluster to the target database.

[0150] Optionally, the device further includes:

[0151] Protocol creation module 605 is used to save the client installation file of the target cluster on the target node of the target database;

[0152] Based on the access information of the target file system of the target cluster, create the configuration file of the target cluster on the target node;

[0153] According to the program in the client installation file, a data sharing protocol is created between the database and the target cluster.

[0154] Optionally, the target database is an MPP database, the target cluster is a Hadoop cluster, and the target file system is a distributed file system HDFS.

[0155] Optionally, the protocol creation module 605 is specifically used for:

[0156] Call the Hadoop cluster client development interface to generate the client installation file;

[0157] Save the client installation file to the target node of the target database.

[0158] Optionally, the protocol creation module 605 is specifically used for:

[0159] The program in the client installation file is invoked to create an external table read function on the target node;

[0160] The program in the client installation file is invoked to create an external table write function on the target node;

[0161] The data sharing protocol is generated based on the read function and the write function;

[0162] Grant the execution rights of the data sharing protocol to database users with preset permissions.

[0163] Optionally, the base processing module 604 is specifically used for:

[0164] Based on the data sharing protocol and the configuration file, the relevant processing of the data sharing instructions is completed using the target external table.

[0165] Optionally, the device further includes:

[0166] Protocol generation module 606 is used to obtain the external protocol of the target database;

[0167] Based on the external protocol, data in external storage is imported into the target database, and / or data in the target database is imported into external storage.

[0168] Optionally, the protocol generation module 606 is specifically used for:

[0169] Receive configuration instructions for the external protocol, the configuration instructions including: protocol type and data format;

[0170] The external protocol is generated based on the protocol type and the data format.

[0171] This invention provides an electronic device, comprising:

[0172] One or more processors;

[0173] Storage device for storing one or more programs.

[0174] When one or more programs are executed by one or more processors, the one or more processors implement the methods of any of the above embodiments.

[0175] This invention provides a computer program product, including a computer program that, when executed by a processor, implements the method of any of the above embodiments of this invention.

[0176] The following is for reference. Figure 7 It shows a schematic diagram of the structure of a computer system 700 suitable for implementing a terminal device of the present invention. Figure 7 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0177] like Figure 7As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the system 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0178] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0179] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs the functions defined above in the system of this invention.

[0180] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0182] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be located in a processor, and for example, can be described as: an instruction receiving module, a table determination module, a protocol acquisition module, and a processing module. The names of these modules do not necessarily limit the module itself; for example, the instruction receiving module can also be described as "a module that receives data sharing instructions for a target database and a target cluster."

[0183] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include:

[0184] Receive data sharing instructions for the target database and target cluster;

[0185] Determine the target external table corresponding to the data sharing instruction, and the target external table is stored on the target cluster;

[0186] Obtain the data sharing protocol between the target database and the target cluster;

[0187] Based on the data sharing protocol, the relevant processing of the data sharing instructions is completed using the target external table.

[0188] According to the technical solution of this embodiment of the invention, the target database is a structured database, such as an MPP database. The target cluster is a cluster using a distributed file system, such as a Hadoop cluster. Through a data sharing protocol, the target database can read and write external tables stored in the target cluster, realizing data sharing between the target database and the target cluster. No intermediate nodes or systems are used for data transfer during the data sharing process. Therefore, the solution of this embodiment of the invention can simply and efficiently realize data sharing between the target database and the target cluster.

[0189] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data sharing method, characterized in that, include: Receive data sharing instructions for the target database and target cluster; The target external table corresponding to the data sharing instruction is determined, and the target external table is stored on the target cluster; wherein, the target external table is an external table of the target database, and the data files of the target external table are stored in the target cluster outside the target database. Different target clusters may correspond to different external tables and data sharing protocols, and the target database may perform read and write operations on different target external tables to complete data sharing with different target clusters. Obtain the data sharing protocol between the target database and the target cluster. The data sharing protocol includes: data transmission protocol, external table read data, external table write function, access information of the target cluster, and permission information. Based on the data sharing protocol, the target external table is used to complete the relevant processing of the data sharing instructions; Before obtaining the data sharing protocol between the target database and the target cluster, the method further includes: saving the client installation file of the target cluster on the target node of the target database; creating a configuration file of the target cluster on the target node according to the access information of the target file system of the target cluster; creating the data sharing protocol between the database and the target cluster according to the program in the client installation file, including: calling the program in the client installation file to create an external table read function on the target node; calling the program in the client installation file to create an external table write function on the target node; generating the data sharing protocol according to the read function and the write function; and granting the execution permission of the data sharing protocol to a database user with preset permissions. The client installation file for the target cluster includes: data transfer functions, external table read functions, external table write functions, access information for the target cluster, and permission information; enabling the data sharing protocol to achieve data sharing between the target database and the target cluster by encapsulating or calling the client installation file for the target cluster. Saving the client installation file to the target node of the target database includes: using the native C++ libhdfs3 as the client to access HDFS, implementing the C interface according to the predetermined API pattern, compiling it into a .so package, and then copying the corresponding .so package to the MPP installation directory; Create a configuration file for the target cluster on the target node, including: creating a data source configuration file, centralizing access information for multiple HDFS file systems, mapping each HDFS with an alias, and configuring the access information of the system under the corresponding alias.

2. The method according to claim 1, characterized in that, The data sharing instruction is used to share data from the target database to the target cluster; Determining the target external table corresponding to the data sharing instruction includes: Based on the data sharing instruction, the storage path of the target external table is determined, and the storage path is located on the target cluster; Based on the data sharing protocol, the target external table is created in the storage path; The process of processing the data sharing instruction based on the data sharing protocol and utilizing the target external table includes: Based on the data sharing protocol, the shared data is written into the target external table using the target database to complete the sharing of data from the target database to the target cluster.

3. The method according to claim 1, characterized in that, The data sharing instruction is used to share data from the target cluster to the target database; The process of processing the data sharing instruction based on the data sharing protocol and utilizing the target external table includes: Based on the data sharing protocol, data is read from the target external table using the target database to complete the sharing of data from the target cluster to the target database.

4. The method according to claim 1, characterized in that, The target database is an MPP database, the target cluster is a Hadoop cluster, and the target file system is a distributed file system HDFS.

5. The method according to claim 4, characterized in that, The step of saving the client installation file of the target cluster on the target node of the target database includes: Call the Hadoop cluster client development interface to generate the client installation file; Save the client installation file to the target node of the target database.

6. The method according to claim 1, characterized in that, The process of processing the data sharing instruction based on the data sharing protocol and utilizing the target external table includes: Based on the data sharing protocol and the configuration file, the relevant processing of the data sharing instructions is completed using the target external table.

7. The method according to claim 1, characterized in that, The method further includes: Obtain the external protocol of the target database; Based on the external protocol, data in external storage is imported into the target database, and / or data in the target database is imported into external storage.

8. The method according to claim 7, characterized in that, Before obtaining the external protocol for the target database, the method further includes: Receive configuration instructions for the external protocol, the configuration instructions including: protocol type and data format; The external protocol is generated based on the protocol type and the data format.

9. A data sharing device, characterized in that, include: The instruction receiving module is used to receive data sharing instructions for the target database and the target cluster; The table determination module is used to determine the target external table corresponding to the data sharing instruction. The target external table is stored on the target cluster. The target external table is an external table of the target database. The data files of the target external table are stored in the target cluster outside the target database. Different target clusters can correspond to different external tables and data sharing protocols. The target database can perform read and write operations on different target external tables to complete data sharing with different target clusters. The protocol acquisition module is used to acquire the data sharing protocol between the target database and the target cluster. The data sharing protocol includes: data transmission protocol, external table read data, external table write function, access information of the target cluster, and permission information. The processing module is used to complete the relevant processing of the data sharing instruction based on the data sharing protocol and the target external table; A protocol creation module is used to store the client installation file of the target cluster on the target node of the target database; create the configuration file of the target cluster on the target node according to the access information of the target file system of the target cluster; create a data sharing protocol between the database and the target cluster according to the program in the client installation file, including: calling the program in the client installation file to create an external table read function on the target node; calling the program in the client installation file to create an external table write function on the target node; generating the data sharing protocol according to the read function and the write function; and granting the execution permission of the data sharing protocol to a database user with preset permissions; the client installation file of the target cluster includes: data transfer function, external table read function, external table write ... The system includes functions for reading external tables, writing external tables, access information for the target cluster, and permission information. It enables data sharing between the target database and the target cluster by encapsulating or calling the target cluster's client installation file. The client installation file is saved to the target node of the target database, including: using the native C++ libhdfs3 as the HDFS access client, implementing the C interface according to a predetermined API pattern, compiling it into a .so package, and then copying the corresponding .so package to the MPP installation directory; creating a configuration file for the target cluster on the target node, including: creating a data source configuration file, centralizing access information for multiple HDFS file systems, mapping each HDFS file system to an alias, and configuring the system's access information under the corresponding alias.

10. The apparatus according to claim 9, characterized in that, The data sharing instruction is used to share data from the target database to the target cluster; The table determination module is specifically used for: Based on the data sharing instruction, the storage path of the target external table is determined, and the storage path is located on the target cluster; Based on the data sharing protocol, the target external table is created in the storage path; The processing module is specifically used for: Based on the data sharing protocol, the shared data is written into the target external table using the target database to complete the sharing of data from the target database to the target cluster.

11. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

12. A computer-readable medium having a computer program stored thereon, characterized in that... When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images