Data processing method and device in multi-cluster scenario, equipment and storage medium

Abstract

This application provides a data processing method, apparatus, device, and storage medium for multi-cluster scenarios. The method involves receiving consumption requests from data consumption applications via a forwarding proxy cluster; forwarding the consumption requests to an access control and data routing service cluster via the forwarding proxy cluster; controlling the access control and data routing service cluster to perform authentication based on the access information in the consumption request; if authentication is successful, controlling the access control and data routing service cluster to query the online transaction processing database to determine the Hadoop cluster Internet Protocol address corresponding to the consumption request, and returning the Hadoop cluster Internet Protocol address to the forwarding proxy cluster; after receiving the Hadoop cluster Internet Protocol address, controlling the forwarding proxy cluster to replace the Internet Protocol address in the consumption request with the Hadoop cluster Internet Protocol address, and sending the data access request to the corresponding Hadoop cluster according to the Hadoop cluster Internet Protocol address, thereby achieving data routing.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, and storage medium for multi-cluster scenarios. Background Technology

[0002] Currently, traditional big data platforms often store data on different Hadoop clusters based on the application and data usage scenarios and frequency. In addition, due to historical system construction and other reasons, multiple Hadoop clusters may also emerge under a big data platform. While independent and physically isolated cluster architectures offer some advantages in improving application efficiency and data security, they also present significant challenges in data access management and data retrieval.

[0003] Currently, there are two main ways to address the above issues. One is to establish a single Hadoop cluster instead of multiple Hadoop clusters. However, a single namespace (Namenode) for managing the file system has significant limitations in large-scale clusters. The other approach is to use a Hadoop federated architecture to combine multiple Hadoop clusters into a unified Hadoop federated cluster.

[0004] However, existing data processing methods suffer from technical problems such as low data access timeliness, high storage costs, and low data processing efficiency. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, and storage medium for multi-cluster scenarios, thereby solving the technical problems of low data access timeliness, high storage cost, and low data processing efficiency in existing data processing methods.

[0006] Firstly, this application provides a data processing method for multi-cluster scenarios, including:

[0007] The data consumption requests sent by the data consumption application are received through the forwarding proxy cluster;

[0008] The consumption request is sent to the access control and data routing service cluster via the forwarding proxy cluster;

[0009] The access control and data routing service cluster performs authentication processing based on the access information in the consumption request;

[0010] If authentication is successful, the access control and data routing service cluster queries the online transaction processing database to determine the Hadoop cluster Internet Protocol address corresponding to the consumption request, and returns the Hadoop cluster Internet Protocol address to the forwarding agent cluster.

[0011] After receiving the Internet Protocol address of the Hadoop cluster, the control forwarding agent cluster replaces the Internet Protocol address in the consumption request with the Internet Protocol address of the Hadoop cluster, and sends the data access request to the corresponding Hadoop cluster according to the Internet Protocol address of the Hadoop cluster, so as to realize data routing.

[0012] This application provides a data routing and access control method for multiple clusters. A data access layer is established around the multiple clusters to control access to downstream systems. At the same time, the underlying details of the big data platform's multiple clusters are shielded, and a unified data access interface is provided to the outside world. A reverse proxy server acts as the external data access interface, which meets the relevant requirements for data access control and data routing under multiple clusters. During the data access process, it is not necessary to know the underlying details such as the namespace where the data is located in advance, and it is not necessary to pre-store a large amount of namespace data, which reduces storage costs. It adopts a non-intrusive design concept, which can be adapted to and compatible with various types of Hadoop clusters, improves the convenience of downstream systems using big data platform data, and improves the timeliness of data access and data processing efficiency.

[0013] Optionally, after the control forwarding agent cluster receives the Internet Protocol address of the Hadoop cluster, replaces the Internet Protocol address in the consumption request with the Hadoop cluster Internet Protocol address, and sends the data access request to the corresponding Hadoop cluster according to the Hadoop cluster Internet Protocol address to achieve data routing, the system further includes:

[0014] After receiving and processing the data access request, the Hadoop cluster sends the relevant returned data to the forwarding agent cluster.

[0015] The control forwarding agent cluster forwards the relevant returned data to the data consumption application.

[0016] Here, this embodiment of the application implements the entire access control and data routing process through a forwarding proxy cluster.

[0017] Optionally, the method further includes:

[0018] Periodically synchronize the Hadoop cluster path list to the online transaction processing database.

[0019] In this embodiment of the application, the path list of each Hadoop cluster is periodically synchronized to the Online Transaction Processing (OLTP) database to realize real-time updates of the path list of each cluster and improve the timeliness of downstream data access.

[0020] Optionally, the periodic synchronization of the Hadoop cluster path list to the online transaction processing database includes:

[0021] Real-time collection of NameNode logs from each Hadoop cluster, and caching of cluster information and log information to Kafka according to a preset storage format;

[0022] Pull cached Hadoop cluster NameNode logs from Kafka in real time and parse out log information involving changes in Hadoop cluster paths;

[0023] If the log information of Hadoop cluster path changes indicates that a new path has been added, then the cluster information and path information will be inserted into the online transaction processing database.

[0024] If the log information of Hadoop cluster path changes indicates path deletion, then the relevant path list record will be deleted by associating and deleting it in the online transaction processing database based on the cluster information and path information.

[0025] Here, in this embodiment of the application, logs from each Hadoop cluster NameNode can be collected in real time and cached in Kafka. Kafka, as a cluster NameNode log caching middleware, is used to cache cluster NameNode logs, thereby improving the stability and accuracy of real-time log processing and further enhancing the accuracy and stability of data processing in multi-cluster scenarios.

[0026] Optionally, the real-time collection of NameNode logs from each Hadoop cluster and the caching of cluster information and log information in Kafka according to a preset storage format includes:

[0027] The Flume tool is used to collect NameNode logs from each Hadoop cluster in real time, and the cluster information and log information are cached in Kafka according to a preset storage format.

[0028] Flume is a highly available, highly reliable, distributed system for collecting, aggregating, and transmitting massive amounts of log data. It supports customizing various data senders within the log system for data collection and is commonly used to interface with Hadoop ecosystem components to collect relevant data. This embodiment selects Flume as the data collection tool for collecting NameNode logs, improving NameNode log collection efficiency and enabling real-time log data collection.

[0029] Optionally, the step of pulling cached Hadoop cluster NameNode logs from Kafka in real time and parsing out log information involving Hadoop cluster path changes includes:

[0030] The Flink tool is used to pull cached Hadoop cluster NameNode logs from Kafka in real time and parse out log information involving changes in Hadoop cluster paths.

[0031] Among them, Flink, compared to other stream computing engines (Storm, Spark Streaming), supports stateful computation, has higher throughput and lower latency, and can handle out-of-order events. This application's embodiment selects Flink as the cluster NameNode log processing engine, parsing NameNode logs in real time and updating file and directory change information to the OLTP database in real time, further improving the efficiency and accuracy of data processing in multi-cluster scenarios.

[0032] Secondly, embodiments of this application provide a data processing apparatus for a multi-cluster scenario, comprising:

[0033] The request receiving module is used to receive consumption requests sent by data consumption applications through the forwarding proxy cluster;

[0034] The first forwarding module is used to send the consumption request to the access control and data routing service cluster through the forwarding proxy cluster;

[0035] The authentication module is used to control the access control and data routing service cluster to perform authentication processing based on the access information in the consumption request;

[0036] The query processing module is used to, if authentication is successful, control access control and data routing service cluster to query the online transaction processing database, determine the Hadoop cluster Internet Protocol address corresponding to the consumption request, and return the Hadoop cluster Internet Protocol address to the forwarding proxy cluster.

[0037] The routing processing module is used to control the forwarding proxy cluster to replace the Internet Protocol address in the consumption request with the Internet Protocol address of the Hadoop cluster after receiving the Internet Protocol address of the Hadoop cluster, and to send the data access request to the corresponding Hadoop cluster according to the Internet Protocol address of the Hadoop cluster, so as to realize data routing.

[0038] Optionally, after the routing processing module forwarding agent cluster receives the Internet Protocol address of the Hadoop cluster, replaces the Internet Protocol address in the consumption request with the Hadoop cluster Internet Protocol address, and sends the data access request to the corresponding Hadoop cluster according to the Hadoop cluster Internet Protocol address to achieve data routing, the above apparatus further includes:

[0039] The request processing module is used to control the Hadoop cluster to send the relevant returned data to the forwarding agent cluster after receiving and processing the data access request.

[0040] The second forwarding module is used to control the forwarding agent cluster to forward the relevant returned data to the data consumption application.

[0041] Optionally, the above-mentioned device further includes:

[0042] The update module is used to periodically synchronize the Hadoop cluster path list to the online transaction processing database.

[0043] Optionally, the update module is specifically used for:

[0044] Real-time collection of NameNode logs from each Hadoop cluster, and caching of cluster information and log information to Kafka according to a preset storage format;

[0045] Pull cached Hadoop cluster NameNode logs from Kafka in real time and parse out log information involving changes in Hadoop cluster paths;

[0046] If the log information of Hadoop cluster path changes indicates that a new path has been added, then the cluster information and path information will be inserted into the online transaction processing database.

[0047] If the log information of Hadoop cluster path changes indicates path deletion, then the relevant path list record will be deleted by associating and deleting it in the online transaction processing database based on the cluster information and path information.

[0048] Optionally, the update module is further specifically used for:

[0049] The Flume tool is used to collect NameNode logs from each Hadoop cluster in real time, and the cluster information and log information are cached in Kafka according to a preset storage format.

[0050] Optionally, the update module is further specifically used for:

[0051] The Flink tool is used to pull cached Hadoop cluster NameNode logs from Kafka in real time and parse out log information involving changes in Hadoop cluster paths.

[0052] Thirdly, this application provides a data processing device for a multi-cluster scenario, comprising: at least one processor and a memory;

[0053] The memory stores computer-executed instructions;

[0054] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the data processing method in a multi-cluster scenario as described in the first aspect and various possible designs of the first aspect.

[0055] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the data processing method in a multi-cluster scenario as described in the first aspect and various possible designs of the first aspect.

[0056] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method in a multi-cluster scenario as described in the first aspect and various possible designs of the first aspect.

[0057] This application provides a data processing method, apparatus, device, and storage medium for multi-cluster scenarios. The method establishes a data access layer around the multi-cluster architecture to control access to downstream systems. It simultaneously shields the underlying details of the multi-cluster big data platform from external layers, providing a unified data access interface. A reverse proxy server acts as the external data access interface, meeting the data access control and routing requirements of multi-cluster environments. During data access, there is no need to know the underlying details such as the namespace where the data resides beforehand, nor is it necessary to pre-store large amounts of namespace data, thus reducing storage costs. Adopting a non-intrusive design, it is compatible with various types of Hadoop clusters, improving the convenience for downstream systems to use data from the big data platform, and enhancing data access timeliness and data processing efficiency. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This application provides a schematic diagram of a data processing system architecture in a multi-cluster scenario.

[0060] Figure 2 This application provides an architecture diagram for an access control and data routing mechanism implementation in a multi-cluster scenario.

[0061] Figure 3 A flowchart illustrating a data processing method in a multi-cluster scenario provided in an embodiment of this application;

[0062] Figure 4 This application provides a schematic diagram of the structure of a real-time synchronization mechanism for the path lists of various Hadoop clusters.

[0063] Figure 5 This is a schematic diagram of the structure of a data processing device in a multi-cluster scenario provided in an embodiment of this application;

[0064] Figure 6 This is a schematic diagram of the structure of a data processing device in a multi-cluster scenario provided in an embodiment of this application.

[0065] The accompanying drawings have illustrated specific embodiments of this disclosure, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0066] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0067] The terms “first,” “second,” “third,” and “fourth,” etc. (if present), in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0068] First, the terms used in the embodiments of this application will be explained:

[0069] Access control: Specifically refers to the verification and control of access requests when downstream systems access data in a multi-cluster environment of a big data platform.

[0070] Data routing: Specifically refers to the ability of a big data platform to automatically route downstream access requests to the corresponding cluster based on the data directory that the downstream system needs to access, thus shielding the downstream system from the details of multiple clusters and lowering the access threshold for the downstream system.

[0071] Hadoop: A distributed system infrastructure. Users can develop distributed programs without understanding the underlying details of distributed systems. It fully leverages the power of clusters for high-speed computing and storage.

[0072] Nginx: is a reverse proxy server.

[0073] Flume is a highly available, highly reliable, distributed system for collecting, aggregating, and transmitting massive amounts of log data. It supports customizing various data senders within the log system for data collection. It is commonly used to integrate with Hadoop ecosystem components to collect relevant data.

[0074] Kafka is a distributed stream processing platform that allows publishing and subscribing to message streams (similar to message queues or enterprise-level messaging systems). It stores message streams in a fault-tolerant and persistent manner and processes messages promptly upon arrival. It is primarily used for data push, large data buffers, log collection, and service middleware.

[0075] Flink is a computing framework and distributed processing engine for stateful computation on unbounded and bounded streams of data. It provides features such as data distribution, data communication, and fault tolerance mechanisms for distributed computation on data streams. Flink offers true real-time stream processing. Compared to other stream processing engines, Flink supports stateful computation, has higher throughput and lower latency, and supports out-of-order event handling.

[0076] Currently, traditional big data platforms often store data on different Hadoop clusters based on the application and data usage scenarios and usage popularity. In addition, due to historical system construction and other reasons, multiple Hadoop clusters may also emerge under the big data platform. Although this independent and physically isolated cluster architecture design has some advantages in improving application running efficiency and data security, it also brings a series of problems, mainly as follows: (1) Data access management is difficult. Because each cluster has its own user system, the big data platform needs to maintain a set of access control policies for downstream systems for each cluster. The management work is repetitive and the workload is large. In addition, when a downstream system accesses a cluster of the big data platform, it also needs to use the corresponding user and password of that cluster for access verification. For the downstream system, the access verification management cost is also very high. (2) Data acquisition is difficult. Because the data is scattered on different independent Hadoop clusters, the downstream system must traverse and access almost all clusters to obtain the required data. Therefore, the downstream system must store the access address and access details of all clusters. Any cluster going online, going offline, or changing may require the downstream system to change the cluster access address. As a result, the coupling between the downstream system and the big data platform is too high, which greatly affects the daily operation and maintenance and future development of the system.

[0077] There are currently two main methods to solve the above problems.

[0078] Method 1 considers establishing a single Hadoop cluster instead of multiple Hadoop clusters. However, a single Namenode has significant limitations in large-scale clusters. In a single Hadoop cluster, the number of files and directories the cluster can handle is directly proportional to the memory size of the Namenode. As the number of files and directories in the cluster grows, it will inevitably reach the Namenode's memory limit. Upgrading or scaling the Namenode requires shutting down the entire Hadoop system, which will inevitably affect the normal operation of high-demand jobs. Method 2 considers using a Hadoop federated architecture to build a unified Hadoop federated cluster from multiple Hadoop clusters. However, after further research, the following drawbacks were found: First, in the federated architecture, the NameNodes are independent of each other, and the NameNode metadata and block files in the DataNodes are not shared between namespaces. If data splitting is required, the DistCp command needs to be used to copy the data completely from one namespace to another, resulting in high storage costs and low copying efficiency. Second, the federated architecture does not provide a unified namespace externally. Each NameNode is independent, and the corresponding namespaces are also independent. In other words, downstream devices still need to know the underlying details such as the namespace where the data is located beforehand, otherwise they will not be able to access the required data.

[0079] Method 1: A single Namenode has significant limitations in large-scale clusters. In a single Hadoop cluster, the number of files and directories the cluster can handle is positively correlated with the memory size of the Namenode. As the number of files and directories in the cluster continues to grow, it will inevitably reach the upper limit of the Namenode's memory capacity. Upgrading and expanding the Namenode requires shutting down the entire Hadoop cluster, which will inevitably affect the normal operation of high-demand jobs.

[0080] Method 2 has the following drawbacks: First, in the federated architecture, NameNodes are independent of each other, and NameNode metadata and block files in DataNodes are not shared between namespaces. If data splitting is required, the DistCp command needs to be used to copy the data completely from one namespace to another, which results in high storage costs and low copying efficiency. Second, the federated architecture does not provide a unified namespace to the outside world. Each NameNode is independent of each other, and the corresponding namespaces are also independent of each other. In other words, downstream devices still need to know the underlying details such as the namespace where the data is located beforehand, otherwise they will not be able to access the required data.

[0081] Therefore, existing technologies suffer from technical problems such as low data access timeliness, high storage costs, and low data processing efficiency.

[0082] To address the aforementioned issues, embodiments of this application provide a data processing method, apparatus, device, and storage medium for multi-cluster scenarios. This method establishes a data access layer around the multi-cluster architecture to control access to downstream systems, while simultaneously shielding the underlying details of the multi-cluster architecture of the big data platform and providing a unified data access interface to the outside world.

[0083] Optional, Figure 1 This is a schematic diagram of a data processing system architecture in a multi-cluster scenario provided by an embodiment of this application. Figure 1 In the above architecture, at least one of data acquisition device 101, processing device 102 and display device 103 is included.

[0084] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the architecture of a data processing system in a multi-cluster scenario. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0085] In the specific implementation process, the data acquisition device 101 may include an input / output interface or a communication interface. The data acquisition device 101 can establish a connection with the bank's network management system or server through the input / output interface or the communication interface to collect the reconciliation data of the user to be reconciled.

[0086] Processing device 102 can establish a data access layer on the periphery of the multi-cluster to control access to downstream systems, while shielding the underlying details of the big data platform multi-cluster and providing a unified data access interface to the outside world.

[0087] Display device 103 can be used to display the above results, and can also interact with users through the display device. Display device 103 can be a user terminal, a user terminal for staff at the branch office awaiting scheduling, or a branch office's network management equipment, server, etc.

[0088] The display device 103 can also be a touch screen, used to receive user commands while displaying the above-mentioned content, so as to realize interaction with the user.

[0089] Figure 2 This application provides an architecture diagram for an access control and data routing mechanism in a multi-cluster scenario, as shown in the embodiments below. Figure 2 As shown, the architecture includes a data consumption application 202, a forwarding proxy cluster 201, an access control pre-data routing service cluster 203, and a Hadoop multi-cluster environment 204. Communication can be achieved between any of these layers.

[0090] The Hadoop multi-cluster environment 204 includes multiple Hadoop clusters. The diagram shows an example with three Hadoop clusters (Hadoop cluster 2041, Hadoop cluster 2042, and Hadoop cluster 2043). In actual applications, there may be more or fewer clusters.

[0091] Each Hadoop cluster consists of Namenode, DateNode, and HTTPFS.

[0092] The forwarding proxy cluster 201 includes multiple reverse proxy servers. The diagram shows an example with three reverse proxy servers (Reverse Proxy Server 2011, Reverse Proxy Server 2012, and Reverse Proxy Server 2013). In practical applications, this can be increased or decreased.

[0093] Among them, the forwarding agent cluster 201, the access control pre-data routing service cluster 203, and the Hadoop multi-cluster environment 204 can be uniformly deployed on the server, and the data consumption application can be the user's terminal device.

[0094] Alternatively, the reverse proxy server can be Nginx.

[0095] Optionally, based on the above architecture, the following functions can be accomplished:

[0096] 1. Use an Nginx cluster as the access interface layer. Data consuming applications send consumption requests to the Nginx cluster.

[0097] 2. The Nginx cluster sends the relevant requests to the access control and data routing service cluster (Web service cluster).

[0098] 3. The access control and data routing service cluster searches the OLTP database for the corresponding record based on the user, password, and file path information in the Uniform Resource Locator (URL) to confirm whether the user has permission to access the requested file (related user permission information can be maintained on the front end). If permission is granted, subsequent operations are performed; otherwise, a verification failure message is returned to the Nginx cluster, which forwards the failure message to the data consumption application, thus implementing access control. If access verification is successful, the access control and data routing service cluster continues to search the OLTP database for the specific Hadoop cluster Internet Protocol (IP) address corresponding to the requested file and returns the Hadoop cluster IP address to the Nginx cluster.

[0099] 4. After receiving the Hadoop cluster IP address, the Nginx cluster replaces the IP address in the user's request URL with the Hadoop cluster IP address and sends the data access request to the corresponding Hadoop cluster, thereby achieving data routing.

[0100] 5. After receiving and processing the access request, the Hadoop cluster sends the relevant return data to the Nginx cluster.

[0101] 6. The Nginx cluster forwards the returned data to the data consuming application, completing the entire access control and data routing process.

[0102] 7. The path list for each Hadoop cluster is periodically synchronized to the OLTP database.

[0103] It should be understood that the aforementioned processing device can be implemented by a processor reading instructions from memory and executing those instructions, or it can be implemented by a chip circuit.

[0104] Furthermore, the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0105] The technical solution of this application will be described in detail below with reference to specific embodiments:

[0106] Optionally, Figure 3 This is a flowchart illustrating a data processing method in a multi-cluster scenario provided by an embodiment of this application. The execution entity of this embodiment can be... Figure 1 The specific execution entity of the processing device 102 can be determined according to the actual application scenario. For example... Figure 3 As shown, the method includes the following steps:

[0107] S301: Receives consumption requests sent by data consumption applications through the forwarding proxy cluster.

[0108] Optionally, the forwarding proxy cluster includes at least one Nginx cluster.

[0109] As a key component in forwarding data access requests, Nginx implements functions such as access request distribution and redirection, and traffic control, thereby further realizing data access control and data routing.

[0110] S302: Send consumption requests to the access control and data routing service cluster via the forwarding proxy cluster.

[0111] S303: The access control and data routing service cluster performs authentication processing based on the access information in the consumption request.

[0112] Optionally, the access information includes at least one of user, password, and file path information.

[0113] Optionally, the access control and data routing service cluster will search the OLTP database for the corresponding record based on the user, password, and file path information in the request URL to confirm whether the user has permission to access the requested file (the relevant user permission information can be maintained on the front end). If the user has permission, the subsequent operation will be performed; if the user does not have permission, a verification failure message will be returned to the Nginx cluster. The Nginx cluster will forward the failure message to the data consumption application, thereby implementing access control. If the access verification is successful, the access control and data routing service cluster will continue to search the OLTP database for the specific Hadoop cluster IP address corresponding to the requested file and return the Hadoop cluster IP address to the Nginx cluster.

[0114] S304: If authentication is successful, the access control and data routing service cluster queries the online transaction processing database to determine the Hadoop cluster Internet Protocol address corresponding to the consumption request, and returns the Hadoop cluster Internet Protocol address to the forwarding agent cluster.

[0115] S305: After receiving the Internet Protocol address of the Hadoop cluster, the control forwarding agent cluster replaces the Internet Protocol address in the consumption request with the Internet Protocol address of the Hadoop cluster, and sends the data access request to the corresponding Hadoop cluster according to the Internet Protocol address of the Hadoop cluster to achieve data routing.

[0116] Optionally, after the control forwarding agent cluster receives the Internet Protocol address of the Hadoop cluster, it replaces the Internet Protocol address in the consumption request with the Internet Protocol address of the Hadoop cluster, and sends the data access request to the corresponding Hadoop cluster according to the Internet Protocol address of the Hadoop cluster to achieve data routing. The system also includes:

[0117] After receiving and processing a data access request, the Hadoop cluster will send the relevant returned data to the forwarding agent cluster.

[0118] Control the forwarding agent cluster to forward relevant returned data to the data consumption application.

[0119] Here, this embodiment of the application implements the entire access control and data routing process through a forwarding proxy cluster.

[0120] This application provides a data routing and access control method for multiple clusters. A data access layer is established around the multiple clusters to control access to downstream systems. Simultaneously, the underlying details of the big data platform's multiple clusters are shielded, providing a unified data access interface. A reverse proxy server acts as the external data access interface, meeting the relevant requirements for data access control and data routing in a multi-cluster environment. During data access, there is no need to know the underlying details such as the namespace where the data resides beforehand, nor is it necessary to pre-store large amounts of namespace data, thus reducing storage costs. Adopting a non-intrusive design, it is adaptable and compatible with various types of Hadoop clusters, improving the convenience for downstream systems to use data from the big data platform, and enhancing data access timeliness and data processing efficiency.

[0121] In one possible implementation, the method also includes:

[0122] Periodically synchronize the Hadoop cluster path list to the online transaction processing database.

[0123] In this embodiment of the application, the path list of each Hadoop cluster is periodically synchronized to the Online Transaction Processing (OLTP) database to realize real-time updates of the path list of each cluster and improve the timeliness of downstream data access.

[0124] Optionally, the Hadoop cluster path list is periodically synchronized to the online transaction processing database, including:

[0125] Real-time collection of NameNode logs from each Hadoop cluster, and caching of cluster information and log information to Kafka according to a preset storage format;

[0126] Pull cached Hadoop cluster NameNode logs from Kafka in real time and parse out log information involving changes in Hadoop cluster paths;

[0127] If the log information of Hadoop cluster path changes indicates that a new path has been added, then the cluster information and path information will be inserted into the online transaction processing database.

[0128] If the log information of Hadoop cluster path changes indicates path deletion, then the relevant path list record will be deleted by associating and deleting it in the online transaction processing database based on the cluster information and path information.

[0129] Here, in this embodiment of the application, logs from each Hadoop cluster NameNode can be collected in real time and cached in Kafka. Kafka, as a cluster NameNode log caching middleware, is used to cache cluster NameNode logs, thereby improving the stability and accuracy of real-time log processing and further enhancing the accuracy and stability of data processing in multi-cluster scenarios.

[0130] Optionally, logs from each Hadoop cluster NameNode are collected in real time, and cluster information and log information are cached in Kafka according to a preset storage format, including:

[0131] The Flume tool is used to collect NameNode logs from each Hadoop cluster in real time, and the cluster information and log information are cached in Kafka according to a preset storage format.

[0132] Flume is a highly available, highly reliable, distributed system for collecting, aggregating, and transmitting massive amounts of log data. It supports customizing various data senders within the log system for data collection and is commonly used to interface with Hadoop ecosystem components to collect relevant data. This embodiment selects Flume as the data collection tool for collecting NameNode logs, improving NameNode log collection efficiency and enabling real-time log data collection.

[0133] Optionally, cached Hadoop cluster NameNode logs are pulled from Kafka in real time, and log information involving Hadoop cluster path changes is parsed out, including:

[0134] The Flink tool is used to pull cached Hadoop cluster NameNode logs from Kafka in real time and parse out log information involving changes in Hadoop cluster paths.

[0135] Among them, Flink, compared to other stream computing engines (Storm, Spark Streaming), supports stateful computation, has higher throughput and lower latency, and can handle out-of-order events. This application's embodiment selects Flink as the cluster NameNode log processing engine, parsing NameNode logs in real time and updating file and directory change information to the OLTP database in real time, further improving the efficiency and accuracy of data processing in multi-cluster scenarios.

[0136] Because the data routing function needs to periodically update the Hadoop cluster path list in the database to meet the needs of visitors querying the corresponding cluster, the update time for the Hadoop cluster path list varies depending on the size of the cluster data. Currently, updating a large Hadoop cluster takes several hours, while updating a smaller Hadoop cluster takes about half an hour. Until the update is complete, the Hadoop cluster path list will retain an older version of its content. To further improve synchronization efficiency, one possible implementation method is... Figure 4 A schematic diagram of the structure of a real-time synchronization mechanism for path lists of various Hadoop clusters provided in this application embodiment ( Figure 4 The number of Hadoop clusters, Flume, Kafka, and Flink in the data is for illustrative purposes only. The real-time synchronization mechanism for the Hadoop cluster path list is as follows:

[0137] 1. Use Flume to collect NameNode logs from each Hadoop cluster in real time, and cache the cluster information and log information in JSON format (e.g., {"CLUSTID":"CLUST_1","LOGS":"specific log information"}) to Kafka to ensure that log information is not lost within a period of time and improve the overall reliability of the system.

[0138] 2. Use Flink to pull cached Hadoop cluster NameNode logs from Kafka in real time and parse out log information related to Hadoop cluster path changes (additions and deletions). For path additions, Flink inserts the cluster information and path information (e.g., ("CLUST_1", "specific path")) into the OLTP database; for path deletions, Flink uses the cluster information and path information (e.g., ("CLUST_1", "specific path")) to associate and delete relevant path list records in the OLTP database.

[0139] The above method can be used to update the path list of each cluster in real time, thereby improving the timeliness of downstream data access.

[0140] This application's implementation of a data access control and data routing mechanism based on tools and technologies such as Nginx, Flume, Kafka, and Flink adopts a non-intrusive design, adaptable to and compatible with various types of Hadoop clusters. This significantly improves the convenience for downstream systems consuming data from the big data platform. Its main advantages are as follows: Compared to Ranger, which can only achieve data access control within a single cluster, this technology achieves access control across multiple clusters, meeting the needs of big data platforms with multiple Hadoop clusters. While recording access, this technology also provides routing control, meeting the data read request distribution requirements of multi-cluster scenarios, shielding the details of multi-cluster access, and simplifying access methods. This technology can effectively manage external access to Hadoop cluster data, recording data access information for subsequent verification and analysis. Based on existing Hadoop clusters, this technology adopts a non-intrusive, loosely coupled architecture design, supporting access control and data routing for underlying heterogeneous clusters. It achieves high-availability data access control in Hadoop multi-cluster scenarios; high-availability data routing in Hadoop multi-cluster scenarios; a high-timeliness and high-availability synchronization mechanism for the path lists of each Hadoop cluster required for data access control and data routing; and the integration of data access control and data routing functions in Hadoop multi-cluster scenarios.

[0141] Figure 5 This is a schematic diagram of the structure of a data processing device in a multi-cluster scenario provided in an embodiment of this application, as shown below. Figure 5As shown, the apparatus in this embodiment includes: a request receiving module 501, a first forwarding module 502, an authentication module 503, a query processing module 504, and a routing processing module 505. The data processing apparatus in this multi-cluster scenario can be the aforementioned processing device itself, a chip or integrated circuit that implements the functions of the processing device, or a server. It should be noted that the division of the request receiving module 501, the first forwarding module 502, the authentication module 503, the query processing module 504, and the routing processing module 505 is only a logical functional division; physically, they can be integrated or independent.

[0142] The request receiving module is used to receive consumption requests sent by data consumption applications through the forwarding proxy cluster.

[0143] The first forwarding module is used to send consumption requests to the access control and data routing service cluster through the forwarding proxy cluster.

[0144] The authentication module is used to control the access control and data routing service cluster to perform authentication processing based on the access information in the consumption request;

[0145] The query processing module is used to query the online transaction processing database if authentication is successful, determine the Hadoop cluster Internet Protocol address corresponding to the consumption request, and return the Hadoop cluster Internet Protocol address to the forwarding agent cluster.

[0146] The routing module controls the forwarding agent cluster to replace the Internet Protocol address in the consumption request with the Hadoop cluster Internet Protocol address after receiving the Hadoop cluster Internet Protocol address, and to send the data access request to the corresponding Hadoop cluster according to the Hadoop cluster Internet Protocol address, so as to realize data routing.

[0147] Optionally, after the routing processing module forwarding agent cluster receives the Hadoop cluster Internet Protocol address, replaces the Internet Protocol address in the consumption request with the Hadoop cluster Internet Protocol address, and sends the data access request to the corresponding Hadoop cluster according to the Hadoop cluster Internet Protocol address to achieve data routing, the above apparatus further includes:

[0148] The request processing module is used to control the Hadoop cluster to send the relevant returned data to the forwarding agent cluster after receiving and processing the data access request.

[0149] The second forwarding module is used to control the forwarding agent cluster to forward the relevant returned data to the data consumption application.

[0150] Optionally, the above-mentioned device further includes:

[0151] The update module is used to periodically synchronize the Hadoop cluster path list to the online transaction processing database.

[0152] Optionally, the update module is specifically used for:

[0153] Real-time collection of NameNode logs from each Hadoop cluster, and caching of cluster information and log information to Kafka according to a preset storage format;

[0154] Pull cached Hadoop cluster NameNode logs from Kafka in real time and parse out log information involving changes in Hadoop cluster paths;

[0155] If the log information of Hadoop cluster path changes indicates that a new path has been added, then the cluster information and path information will be inserted into the online transaction processing database.

[0156] If the log information of Hadoop cluster path changes indicates path deletion, then the relevant path list record will be deleted by associating and deleting it in the online transaction processing database based on the cluster information and path information.

[0157] Optionally, the update module is also specifically used for:

[0158] The Flume tool is used to collect NameNode logs from each Hadoop cluster in real time, and the cluster information and log information are cached in Kafka according to a preset storage format.

[0159] Optionally, the update module is also specifically used for:

[0160] The Flink tool is used to pull cached Hadoop cluster NameNode logs from Kafka in real time and parse out log information involving changes in Hadoop cluster paths.

[0161] refer to Figure 6The diagram illustrates a structural schematic of a data processing device 600 suitable for implementing embodiments of the present disclosure in a multi-cluster scenario. This data processing device 600 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The data processing device shown in the multi-cluster scenario is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments disclosed herein.

[0162] like Figure 6 As shown, the data processing device 600 in a multi-cluster scenario may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the data processing device 600 in a multi-cluster scenario. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0163] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows data processing equipment 600 in multi-cluster scenarios to communicate wirelessly or wiredly with other devices to exchange data. Although... Figure 6 A data processing device 600 with various devices in a multi-cluster scenario is shown; however, it should be understood that implementation or having all of the devices shown is not required. More or fewer devices may be implemented alternatively.

[0164] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0165] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0166] The aforementioned computer-readable medium may be included in the data processing device in the aforementioned multi-cluster scenario; or it may exist independently and not be assembled into the data processing device in the multi-cluster scenario.

[0167] The aforementioned computer-readable medium carries one or more programs, which, when executed by the data processing device in the multi-cluster scenario, cause the data processing device in the multi-cluster scenario to perform the method shown in the above embodiments.

[0168] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0170] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0171] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0172] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0173] The data processing device in the multi-cluster scenario of this application embodiment can be used to execute the technical solutions in the above-described method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0174] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the above-described data processing methods.

[0175] This application also provides a computer program product, including a computer program, which, when executed by a processor, is used to implement any of the above-described data processing methods.

[0176] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0177] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0178] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0179] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data processing method in a multi-cluster scenario, characterized in that, include: The data consumption requests sent by the data consumption application are received through the forwarding proxy cluster; The consumption request is sent to the access control and data routing service cluster via the forwarding proxy cluster; The access control and data routing service cluster performs authentication processing based on the access information in the consumption request; If authentication is successful, the access control and data routing service cluster queries the online transaction processing database to determine the Hadoop cluster Internet Protocol address corresponding to the consumption request, and returns the Hadoop cluster Internet Protocol address to the forwarding agent cluster. After receiving the Internet Protocol address of the Hadoop cluster, the control forwarding agent cluster replaces the Internet Protocol address in the consumption request with the Internet Protocol address of the Hadoop cluster, and sends the data access request to the corresponding Hadoop cluster according to the Internet Protocol address of the Hadoop cluster, so as to realize data routing. The method further includes: Real-time collection of NameNode logs from each Hadoop cluster, and caching of cluster information and log information to Kafka according to a preset storage format; Pull cached Hadoop cluster NameNode logs from Kafka in real time and parse out log information involving changes in Hadoop cluster paths; If the log information of Hadoop cluster path changes indicates that a new path has been added, then the cluster information and path information will be inserted into the online transaction processing database. If the log information of Hadoop cluster path changes indicates path deletion, then the relevant path list record will be deleted by associating and deleting it in the online transaction processing database based on the cluster information and path information.

2. The method of claim 1, wherein, After the control forwarding agent cluster receives the Internet Protocol address of the Hadoop cluster, it replaces the Internet Protocol address in the consumption request with the Hadoop cluster Internet Protocol address, and sends the data access request to the corresponding Hadoop cluster according to the Hadoop cluster Internet Protocol address to achieve data routing. The process further includes: After receiving and processing the data access request, the Hadoop cluster sends the relevant returned data to the forwarding agent cluster. The control forwarding agent cluster forwards the relevant returned data to the data consumption application.

3. The method of claim 1, wherein, The process involves real-time collection of NameNode logs from each Hadoop cluster and caching the cluster information and log information in a preset storage format to Kafka, including: The Flume tool is used to collect NameNode logs from each Hadoop cluster in real time, and the cluster information and log information are cached in Kafka according to a preset storage format.

4. The method of claim 1, wherein, The method involves real-time retrieval of cached Hadoop cluster NameNode logs from Kafka and parsing log information related to Hadoop cluster path changes, including: The Flink tool is used to pull cached Hadoop cluster NameNode logs from Kafka in real time and parse out log information involving changes in Hadoop cluster paths.

5. A data processing apparatus in a multi-cluster scenario, the apparatus comprising: include: The request receiving module is used to receive consumption requests sent by data consumption applications through the forwarding proxy cluster; The first forwarding module is used to send the consumption request to the access control and data routing service cluster through the forwarding proxy cluster; The authentication module is used to control the access control and data routing service cluster to perform authentication processing based on the access information in the consumption request; The query processing module is used to, if authentication is successful, control access control and data routing service cluster to query the online transaction processing database, determine the Hadoop cluster Internet Protocol address corresponding to the consumption request, and return the Hadoop cluster Internet Protocol address to the forwarding proxy cluster. The routing processing module is used to control the forwarding proxy cluster to replace the Internet Protocol address in the consumption request with the Internet Protocol address of the Hadoop cluster after receiving the Internet Protocol address of the Hadoop cluster, and to send the data access request to the corresponding Hadoop cluster according to the Internet Protocol address of the Hadoop cluster, so as to realize data routing. The update module is used to collect NameNode logs from each Hadoop cluster in real time and cache cluster information and log information in Kafka according to a preset storage format; The system pulls cached Hadoop cluster NameNode logs from Kafka in real time and parses out log information related to Hadoop cluster path changes. If the Hadoop cluster path change log information indicates a path addition, the system inserts the cluster information and path information into the online transaction processing database. If the Hadoop cluster path change log information indicates a path deletion, the system associates and deletes the relevant path list records in the online transaction processing database based on the cluster information and path information.

6. A data processing device for multi-cluster scenarios, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the data processing method in a multi-cluster scenario as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data processing method in a multi-cluster scenario as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 4.

Images