Data interaction method and device for Hadoop cluster and Greenplum database

Abstract

The invention provides a data interaction method and device for a Hadoop cluster and a Greenpage database, and the method comprises the steps: configuring a Presto client on a submitting machine of a Spark task, and connecting to the Greenpage database through the Presto client; the structure information of the target data table in the Greenplum database is inquired through the Presto client side; creating a data table of which the table structure is the same as that of the target data table in Hive; querying target data in the target data table through the Presto client; inserting the target data into a data table created in the Hive; and the Spark task reads the target data inserted in the data table created in the Hive. According to the method and the device, the problems of security risk and complex management of access to the Greenplum database by the Hadoop cluster in related technologies are solved, unauthorized access to the Greenplum database by the Hadoop cluster is realized, the access flexibility of the Greenplum data by the Spark is improved, and the data security is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of big data analysis, and in particular to a method and device for data interaction between a Hadoop cluster and a Greenplum database. Background Art

[0002] In the field of big data processing, Hadoop and Spark are two widely used technologies that provide powerful distributed computing capabilities. Greenplum Database, as a high-performance, large-scale parallel processing database system, is often used to process a large number of data analysis tasks. Related technologies usually use the following two methods to implement Hadoop cluster access to Greenplum Database:

[0003] 1. Direct access permission: Grant Greenplum database access permissions to nodes in the Hadoop cluster. In this way, Spark tasks can directly connect to the Greenplum database to read or write data for analysis.

[0004] 2. Data export: Export the data in the Greenplum database to a file format such as CSV or Parquet, and then upload these files to the Hadoop cluster. Spark tasks can then process these files on the Hadoop cluster.

[0005] The above method has the following technical defects:

[0006] 1. Security risks: Granting direct access to the Greenplum database to Hadoop cluster nodes may pose security risks. Each node may become a potential security vulnerability, increasing the risk of data leakage or unauthorized access.

[0007] 2. Management complexity: The direct access permission method requires additional management work, including permission configuration, monitoring, and auditing. This increases the complexity of system management and may lead to increased operation and maintenance costs.

[0008] 3. Performance bottleneck: Data export methods may encounter performance bottlenecks, especially when processing large-scale data sets. The export and upload process may consume a lot of time and resources, affecting the overall data processing efficiency.

[0009] 4. Data Privacy: Direct access rights methods may violate data privacy protection principles, especially when sensitive or protected data is involved. This may lead to compliance issues and possible violations of data protection regulations.

[0010] Currently, there are security risks and complex management issues in Hadoop cluster accessing Greenplum database in related technologies, and no effective solution has been proposed. Summary of the invention

[0011] The purpose of the present disclosure is to address the deficiencies in the prior art and to provide a method, device, electronic device, and computer-readable storage medium for data interaction between a Hadoop cluster and a Greenplum database, so as to at least solve the problems in the related art of security risks and complex management of Hadoop cluster accessing a Greenplum database.

[0012] According to one aspect of the present disclosure, a method for data interaction between a Hadoop cluster and a Greenplum database is provided, comprising:

[0013] Configure the Presto client on the Spark task submission machine and connect to the Greenplum database through the Presto client.

[0014] Querying the structure information of the target data table in the Greenplum database through the Presto client;

[0015] Create a data table in Hive that has the same table structure as the target data table;

[0016] Query the target data in the target data table through the Presto client;

[0017] Insert the target data into the data table created in Hive;

[0018] The Spark task reads the target data inserted into the data table created in Hive.

[0019] According to another aspect of the present disclosure, a data interaction device between a Hadoop cluster and a Greenplum database is provided, comprising:

[0020] A connection unit, used to configure a Presto client on a Spark task submission machine and connect to a Greenplum database through the Presto client;

[0021] A first query unit, configured to query structural information of a target data table in the Greenplum database through the Presto client;

[0022] A creation unit, used to create a data table having the same table structure as the target data table in Hive;

[0023] A second query unit, configured to query the target data in the target data table through the Presto client;

[0024] An insert unit, used to insert the target data into a data table created in Hive;

[0025] A reading unit is used for the Spark task to read the target data inserted into the data table created in Hive.

[0026] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0027] Processor; and

[0028] Memory for storing programs,

[0029] The program includes instructions, which, when executed by the processor, enable the processor to execute the data interaction method between the Hadoop cluster and the Greenplum database in the present disclosure.

[0030] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the data interaction method between the Hadoop cluster and the Greenplum database in the present disclosure.

[0031] One or more technical solutions provided in the embodiments of the present disclosure configure a Presto client on the submitting machine of the Spark task, and connect to the Greenplum database through the Presto client; query the structural information of the target data table in the Greenplum database through the Presto client; create a data table with the same table structure as the target data table in Hive; query the target data in the target data table through the Presto client; insert the target data into the data table created in Hive; the Spark task reads the target data inserted in the data table created in Hive. The Spark task can indirectly analyze the data in the Greenplum database by accessing the data in Hive without directly connecting to the Greenplum database, thereby avoiding the restrictions of security policies and ensuring that the needs of data analysis are met. It can solve the problems of security risks and complex management in the Hadoop cluster accessing the Greenplum database in the related technology, realize the unauthorized access of the Hadoop cluster to the Greenplum database, improve the access flexibility of Spark to Greenplum data, and ensure data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0033] Figure 1 A schematic diagram of the architecture of a data interaction system between a Hadoop cluster and a Greenplum database according to an exemplary embodiment of the present disclosure is shown;

[0034] Figure 2 A schematic diagram showing a data interaction process between a Hadoop cluster and a Greenplum database without direct access rights according to an exemplary embodiment of the present disclosure;

[0035] Figure 3 A flowchart of a method for data interaction between a Hadoop cluster and a Greenplum database according to an exemplary embodiment of the present disclosure is shown;

[0036] Figure 4 A schematic block diagram of a data interaction device between a Hadoop cluster and a Greenplum database according to an exemplary embodiment of the present disclosure is shown;

[0037] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0038] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0039] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0040] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0041] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0042] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0043] Aspects of the present disclosure are described below with reference to the accompanying drawings.

[0044] First, the technical terms in this application are explained as follows:

[0045] Greenplum: An open source massively parallel processing (MPP) database server based on PostgreSQL, designed for data warehouses and large-scale data analysis. It provides high-performance data query and analysis capabilities through parallel processing and data distribution technology. Greenplum supports the SQL query language and can process PB-level data.

[0046] Presto: An open source distributed SQL query engine for high-performance, large-scale data queries. It can connect to a variety of data sources, including Hive, MySQL, PostgreSQL, Greenplum, etc., and supports joint queries across multiple data sources. Presto is designed to respond to queries quickly and be able to handle large-scale data sets.

[0047] Hive: A data warehouse infrastructure built on Hadoop, developed by Facebook to provide data summarization, query, and analysis. It allows users to query big data stored in the Hadoop Distributed File System (HDFS) using a SQL-like query language (called HiveQL). Hive is suitable for batch processing and big data analysis, rather than real-time queries.

[0048] Hadoop: An open source framework, developed by the Apache Software Foundation, that allows distributed processing of large-scale data sets. It consists of two main components: the Hadoop Distributed File System (HDFS), which is used to store big data; and HadoopYARN, a resource management platform for managing computing resources and running applications on clusters. Hadoop is often used with MapReduce, a programming model for large-scale data processing.

[0049] Spark: An open source distributed computing system developed by UC Berkeley AMP Lab and later became part of the Apache Software Foundation. It provides a fast, general, and easy-to-use big data processing platform. Spark supports multiple programming languages, including Java, Scala, Python, and R, and provides high-level APIs to simplify the implementation of big data analysis tasks.

[0050] In the field of big data processing, Hadoop and Spark are two widely used technologies that provide powerful distributed computing capabilities. As a high-performance, large-scale parallel processing database system, Greenplum Database is often used to handle a large number of data analysis tasks. However, in actual applications, due to security policy restrictions, Hadoop clusters may not be able to directly access Greenplum Database, which brings challenges to data analysis.

[0051] Based on the above problems, the present invention provides a data interaction system, method and device between a Hadoop cluster and a Greenplum database, aiming to solve the problem that the Hadoop cluster has no permission to directly access the Greenplum database, while ensuring the security of the data and the stability of the system. Through the present invention, the data in the Greenplum database can be safely and indirectly analyzed by using the distributed computing capability of Spark without violating the security policy.

[0052] The exemplary embodiments of the present disclosure provide a data interaction system between a Hadoop cluster and a Greenplum database. Figure 1 FIG. 1 shows a schematic diagram of the architecture of a data interaction system between a Hadoop cluster and a Greenplum database according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the system includes: Greenplum database, Spark tasks running in the Hadoop cluster, Presto clients configured on the Spark task submission machine, and Hive.

[0053] Figure 2A schematic diagram of a data interaction process between a Hadoop cluster and a Greenplum database without direct access rights according to an exemplary embodiment of the present disclosure is shown. Figure 2 As shown in the figure, the process specifically includes the following key steps:

[0054] 1. Connect to Greenplum data source: First, on the machine submitting the Spark task, use the Presto client to connect to the Greenplum database using appropriate connection parameters. Presto is an efficient distributed SQL query engine that supports connections to a variety of data sources, including but not limited to Greenplum databases. The Presto client allows efficient access to the Greenplum database.

[0055] 2. Query the Greenplum table structure: Use the Presto client to query the structure information of the target data table in the Greenplum database, including the table name, field name, field type, etc. This step is to create a table with the same structure as the Greenplum table on Hive in the subsequent steps, so as to achieve seamless data migration and access.

[0056] 3. Build a Hive table: Create a table in Hive with the same table structure as the target data table in the Greenplum database. Hive is a data warehouse infrastructure built on Hadoop. It supports HiveQL, a query language similar to SQL, and is suitable for processing large data sets. When creating a Hive table, you need to specify the table name, field name, field type, etc. based on the table structure information previously queried to ensure that the structure of the table created in Hive is consistent with the target data table in the Greenplum database.

[0057] 4. Query Greenplum data: Execute SQL queries through the Presto client to retrieve all the data in the target data table from the Greenplum database. This step uses the efficient query capabilities of Presto to quickly obtain the data stored in the target data table in the Greenplum database.

[0058] 5. Insert data into Hive table: Insert the query data into the table created in Hive. This can be achieved by executing the INSERT INTO statement in the Presto client, using the query results as the data source and inserting them into the Hive table. This step completes the migration of data from Greenplum Database to Hive, providing a basis for subsequent data analysis.

[0059] 6. Data analysis: The Spark task analyzes the data read from the Hive table to obtain the data analysis results.

[0060] Through the above steps, Spark tasks can indirectly analyze data in the Greenplum database by accessing data in Hive without directly connecting to the Greenplum database, thereby complying with security policy restrictions and ensuring that data analysis needs are met.

[0061] The technical solution of the exemplary embodiment of the present disclosure has the following advantages:

[0062] 1) Improved security: Through indirect access, potential security risks brought by direct authorization are avoided, and the security of data access is enhanced.

[0063] 2) Simplified management: It reduces the need to configure and manage direct database access rights for Hadoop cluster nodes, reducing the complexity of system management and operation and maintenance costs.

[0064] 3) Privacy protection: Ensure that data privacy protection principles are followed during data transmission and access, reducing the risk of compliance issues and violations of data protection regulations. By implementing strict data privacy protection measures during data access, it ensures that sensitive and protected data will not be accessed without authorization, thereby reducing the risk of data leakage.

[0065] 4) Enhanced flexibility: Increased flexibility in data access enables Spark tasks to access data in the Greenplum database as needed without being restricted by direct access permissions. This flexibility is particularly important for scenarios that require rapid iteration and adjustment of data analysis tasks.

[0066] An exemplary embodiment of the present disclosure provides a method for data interaction between a Hadoop cluster and a Greenplum database. Figure 3 A flow chart of a method for data interaction between a Hadoop cluster and a Greenplum database according to an exemplary embodiment of the present disclosure is shown. Figure 3 As shown, the method comprises the following steps:

[0067] Step S301, configuring a Presto client on the Spark task submission machine, and connecting to the Greenplum database through the Presto client;

[0068] Step S302, querying the structure information of the target data table in the Greenplum database through the Presto client;

[0069] Step S303, creating a data table with the same table structure as the target data table in Hive;

[0070] Step S304, querying the target data in the target data table through the Presto client;

[0071] Step S305, inserting the target data into the data table created in Hive;

[0072] Step S306: The Spark task reads the target data inserted into the data table created in Hive.

[0073] Through the above steps, an intermediary data access method can be implemented. By leveraging the capabilities of the Presto client and Hive, a secure and flexible data access solution is provided for the Hadoop cluster. This solution not only solves the problem of direct access permission restrictions, but also improves the security and flexibility of data access, providing new possibilities for big data analysis.

[0074] In some embodiments, the step S303 creates a data table having the same table structure as the target data table in Hive, including:

[0075] Create a data table in Hive that has the same table structure as the target data table;

[0076] Specify the table name, field name, and field type of the data table created in Hive.

[0077] This embodiment can achieve seamless data migration and access by creating a table with the same structure as the Greenplum table on Hive.

[0078] In some embodiments, the step S304 queries the target data in the target data table through the Presto client, including:

[0079] Detect whether the type of the target data in the target data table is NUMERIC type;

[0080] If it is detected that the type of the target data is a NUMERIC type, the target data is type-converted to a DOUBLE type.

[0081] Because Presto does not support high-precision fields, high-precision fields need to be automatically detected and converted during query. Therefore, this embodiment automatically detects high-precision fields in the target data table in the Greenplum database and performs type conversion during query to convert the high-precision NUMERIC type to the DOUBLE type, thereby improving the accuracy of data access.

[0082] In some embodiments, after the Spark task in step S306 reads the target data inserted into the data table created in Hive, the method further includes:

[0083] The Spark task performs data analysis on the target data to obtain a data analysis result;

[0084] Output and display the data analysis results.

[0085] This embodiment uses Spark tasks to read data from Hive tables, performs data analysis on the read data, and outputs and displays the data analysis results, which can improve the readability of data analysis and thus improve the user experience. It should be noted that this embodiment does not specifically limit the display method of data analysis results, and can be set to charts, tables, etc. according to actual needs.

[0086] When a Spark task in a Hadoop cluster needs to read Greenplum database resources, but the Hadoop cluster lacks direct access rights due to security policy restrictions, the present invention provides a solution, through which the Hadoop cluster can indirectly and securely access the Greenplum database without violating permission management, thereby ensuring data availability and system security.

[0087] The present invention can solve the following technical problems:

[0088] 1. Solution to security issues: In the existing data processing environment, direct access to the database is often accompanied by security risks. Especially in scenarios involving high-value data, how to perform effective data access and analysis while ensuring data security has become an urgent problem to be solved. The present invention introduces an intermediary data access method, which enables data processing tasks to indirectly access and analyze data without directly accessing the target database, thereby significantly reducing the risk of data leakage and unauthorized access and enhancing the security of the entire data processing process.

[0089] 2. Reduction of management complexity: With the increase of data processing tasks and the diversification of data sources, it becomes increasingly complex to configure and manage direct database access rights for each data processing node, which not only increases the difficulty of system management, but also increases the operation and maintenance costs. The solution provided by the present invention simplifies the data access management process by reducing the configuration requirements of direct database access rights, reduces the management complexity and operation and maintenance costs of the system, and improves the overall efficiency and maintainability of the data processing system.

[0090] 3. Strengthening data privacy protection: Data processing under the premise of complying with the data privacy protection principle has become a basic requirement for modern data applications. Especially for databases containing sensitive information and protected data, how to protect data privacy while analyzing data has become an important technical challenge. The present invention ensures the privacy protection of data during transmission and access by implementing an indirect data access mechanism, reduces the compliance issues and data protection regulations that may be caused by data processing activities, and provides strong technical support for the secure processing of sensitive data.

[0091] 4. Improved data access flexibility: In traditional data processing models, data access is often restricted by direct access permissions, which limits the flexibility and efficiency of data processing tasks. The present invention provides a new data access method that allows data processing tasks to flexibly access data in the target database according to actual needs without worrying about the constraints of direct access permissions. This enhanced flexibility not only improves the efficiency of data processing, but also provides more possibilities for complex data analysis tasks.

[0092] In summary, the present invention effectively solves the technical problems in ensuring data security, simplifying management processes, strengthening data privacy protection, and improving data access flexibility by proposing a new intermediary data access method and system, and provides a new solution for data processing in the big data era.

[0093] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0094] The exemplary embodiments of the present disclosure also provide a data interaction device between a Hadoop cluster and a Greenplum database, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the terms "module", "unit", "subunit", etc. may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0095] Figure 4 A schematic block diagram of a data interaction device between a Hadoop cluster and a Greenplum database according to an exemplary embodiment of the present disclosure is shown. Figure 4 As shown, the device comprises:

[0096] A connection unit 41 is used to configure a Presto client on a Spark task submission machine and connect to a Greenplum database through the Presto client;

[0097] A first query unit 42 is used to query the structure information of the target data table in the Greenplum database through the Presto client;

[0098] A creation unit 43 is used to create a data table having the same table structure as the target data table in Hive;

[0099] A second query unit 44 is used to query the target data in the target data table through the Presto client;

[0100] An inserting unit 45, used for inserting the target data into a data table created in Hive;

[0101] The reading unit 46 is used for the Spark task to read the target data inserted into the data table created in Hive.

[0102] In some embodiments, the second query unit 44 includes:

[0103] A detection module, used for detecting whether the type of the target data in the target data table is a NUMERIC type;

[0104] The conversion module is used to perform type conversion on the target data if it is detected that the type of the target data is a NUMERIC type, and convert the type of the target data into a DOUBLE type.

[0105] In some embodiments, the creating unit 43 includes:

[0106] A creation module is used to create a data table with the same table structure as the target data table in Hive;

[0107] Specify the module, which is used to specify the table name, field name, and field type of the data table created in Hive.

[0108] In some embodiments, the device further comprises:

[0109] an analysis unit, configured to, after the Spark task reads the target data inserted into the data table created in Hive, perform data analysis on the target data by the Spark task to obtain a data analysis result;

[0110] The output unit is used to output and display the data analysis results.

[0111] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0112] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0113] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0114] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.

[0115] refer to Figure 5 , a block diagram of an electronic device 500 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0116] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0117] A plurality of components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 may be any type of device capable of inputting information to the electronic device 500, and the input unit 506 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 507 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 may include but is not limited to a disk, an optical disk. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0118] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the data interaction method between the Hadoop cluster and the Greenplum database may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. In some embodiments, the computing unit 501 may be configured to execute the data interaction method between the Hadoop cluster and the Greenplum database in any other appropriate manner (e.g., by means of firmware).

[0119] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0120] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0121] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0124] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A data interaction method between a Hadoop cluster and a Greenplum database, characterized in that: include: Configure the Presto client on the Spark task submission machine and connect to the Greenplum database through the Presto client. Querying the structure information of the target data table in the Greenplum database through the Presto client; Create a data table in Hive that has the same table structure as the target data table; Query the target data in the target data table through the Presto client; Insert the target data into the data table created in Hive; The Spark task reads the target data inserted into the data table created in Hive.

2. The method according to claim 1, characterized in that The querying the target data in the target data table through the Presto client includes: Detect whether the type of the target data in the target data table is NUMERIC type; If it is detected that the type of the target data is a NUMERIC type, the target data is type-converted to a DOUBLE type.

3. The method according to claim 1, characterized in that The step of creating a data table having the same table structure as the target data table in Hive includes: Create a data table in Hive that has the same table structure as the target data table; Specify the table name, field name, and field type of the data table created in Hive.

4. The method according to any one of claims 1 to 3, characterized in that After the Spark task reads the target data inserted into the data table created in Hive, the method further includes: The Spark task performs data analysis on the target data to obtain a data analysis result; Output and display the data analysis results.

5. A data interaction device between a Hadoop cluster and a Greenplum database, characterized in that: include: A connection unit, used to configure a Presto client on a Spark task submission machine and connect to a Greenplum database through the Presto client; A first query unit, configured to query structural information of a target data table in the Greenplum database through the Presto client; A creation unit, used to create a data table having the same table structure as the target data table in Hive; A second query unit, configured to query the target data in the target data table through the Presto client; An insert unit, used to insert the target data into a data table created in Hive; A reading unit is used for the Spark task to read the target data inserted into the data table created in Hive.

6. The device according to claim 5, characterized in that The second query unit includes: A detection module, used for detecting whether the type of the target data in the target data table is a NUMERIC type; The conversion module is used to perform type conversion on the target data if it is detected that the type of the target data is a NUMERIC type, and convert the type of the target data into a DOUBLE type.

7. The device according to claim 5, characterized in that The creation unit comprises: A creation module is used to create a data table with the same table structure as the target data table in Hive; Specify the module, which is used to specify the table name, field name, and field type of the data table created in Hive.

8. The device according to any one of claims 5 to 7, characterized in that Also includes: an analysis unit, configured to, after the Spark task reads the target data inserted into the data table created in Hive, perform data analysis on the target data by the Spark task to obtain a data analysis result; The output unit is used to output and display the data analysis results.

9. An electronic device, characterized in that: include: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 4.

Images