Aggregation access methods, devices, equipment, storage media, and products for distributed clusters

Abstract

This application discloses a method, apparatus, device, storage medium, and product for aggregated access to distributed clusters, relating to the field of data access technology. The method includes: aggregating multiple distributed HDFS clusters based on a unified namespace to obtain an aggregated cluster; acquiring aggregated access requests from HDFS clients to the aggregated cluster, parsing the aggregated access requests, and determining the target data, target node, and request type of the aggregated access requests; and routing the aggregated access requests to the corresponding target HDFS cluster within the aggregated cluster based on the target data, target node, and request type of the aggregated access requests, and accessing the data blocks in the target HDFS cluster. Through this method, aggregating widely distributed HDFS clusters using a unified namespace enables HDFS clients to easily and uniformly access data in each cluster, and achieves data interaction through intelligent request analysis and routing mechanisms.

Description

Technical Field

[0001] This application relates to the field of data access technology, and in particular to aggregation access methods, apparatus, devices, storage media and products for distributed clusters. Background Technology

[0002] Currently, access to HDFS (Hadoop Distributed File System) clusters typically employs a combination of the HDFS federation mechanism and the ViewFS (View File System) mechanism. This significantly increases the scalability and fault tolerance of a single HDFS cluster and provides a simple, unified access view for HDFS clients. However, the ViewFS mechanism requires configuration on all clients, making management complex. Different clients require different configurations, and adding a new client or user directory necessitates a complete configuration overhaul, increasing the management burden and complexity of later maintenance. Furthermore, it still focuses on a single HDFS cluster and cannot achieve unified access management for multiple widely distributed HDFS clusters.

[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a method, apparatus, device, storage medium and product for aggregated access of distributed clusters, aiming to solve the technical problem that traditional methods in the prior art are difficult to achieve unified access management of multiple widely distributed HDFS clusters.

[0005] To achieve the above objectives, this application provides a method for aggregated access of distributed clusters, the method comprising:

[0006] Based on a unified namespace, multiple scattered HDFS clusters are aggregated to obtain an aggregated cluster.

[0007] Obtain the aggregation access request from the HDFS client to the aggregation cluster, parse the aggregation access request, and determine the target data, target node, and request type of the aggregation access request;

[0008] Based on the target data, target node, and request type of the aggregated access request, the aggregated access request is routed to the corresponding target HDFS cluster in the aggregated cluster to access the data blocks in the target HDFS cluster.

[0009] In one embodiment, the step of routing the aggregated access request to the corresponding target HDFS cluster in the aggregated cluster based on the target data, target node, and request type of the aggregated access request, and accessing the data blocks in the target HDFS cluster, includes:

[0010] When the target node of the aggregated access request is a metadata management node, the physical address of the target HDFS cluster corresponding to the target data is determined based on the logical address of the target data in the aggregated access request, and the aggregated access request is adjusted accordingly.

[0011] Based on the physical address of the target HDFS cluster, the adjusted aggregated access request is routed to the metadata management node of the corresponding target HDFS cluster to access the data blocks in the metadata management node of the target HDFS cluster.

[0012] In one embodiment, the step of determining the physical address of the target HDFS cluster corresponding to the target data based on the logical address of the target data includes:

[0013] Obtain the address mapping table of the aggregation cluster, which stores the physical address of the HDFS cluster and the logical address mapped from the physical address of the HDFS cluster.

[0014] Based on the logical address of the target data, determine the physical address that matches the logical address of the target data in the address mapping table;

[0015] The HDFS cluster corresponding to the matched physical address is taken as the target HDFS cluster corresponding to the target data, and the physical address of the target HDFS cluster is obtained.

[0016] In one embodiment, the step of routing the aggregated access request to the corresponding target HDFS cluster in the aggregated cluster based on the target data, target node, and request type of the aggregated access request, and accessing the data blocks in the target HDFS cluster, includes:

[0017] When the target node of the aggregated access request is a data storage node, obtain the request return information of the metadata management node in the aggregated cluster;

[0018] Based on the request return information, determine the target data block of the target data of the aggregated access request in the target node and the physical address of the target node corresponding to the target data block;

[0019] When the request type of the aggregated access request is a write request, the target HDFS cluster of the aggregated access request is determined based on the physical address of the target node, and the aggregated access request is routed to the corresponding data storage node in the target HDFS cluster to access the target data block.

[0020] In one embodiment, after the step of determining the target data block of the aggregated access request in the target node and the physical address of the target node corresponding to the target data block based on the request return information, the method further includes:

[0021] When the request type of the aggregated access request is a read request, the cached data block of the aggregated cluster is obtained;

[0022] When the target data block corresponding to the aggregated access request matches the cached data block, the aggregated access request is determined to be a cache hit read request. Based on the cache address of the cached data block, the cache hit read request is routed to the corresponding cached data block and the cached data block is accessed.

[0023] When the target data block corresponding to the aggregated access request does not match the cached data block, the aggregated access request is determined to be a non-cache hit read request. Based on the physical address of the target node, the target HDFS cluster of the cache hit read request is determined, and the non-cache hit read request is routed to the corresponding data storage node in the target HDFS cluster to access the target data block.

[0024] In one embodiment, the method further includes:

[0025] Obtain the status of data blocks on the data storage nodes in the aggregation cluster;

[0026] Data blocks in the fully written state are selected as candidate data blocks;

[0027] Based on the data popularity of the candidate data blocks, the cached data blocks of the aggregation cluster are determined.

[0028] Furthermore, to achieve the above objectives, this application also proposes an aggregated access device for a distributed cluster, the aggregated access device for the distributed cluster comprising:

[0029] The aggregation module is used to aggregate multiple scattered HDFS clusters based on a unified namespace to obtain an aggregated cluster.

[0030] The access module is used to obtain the aggregation access request from the HDFS client to the aggregation cluster, parse the aggregation access request, and determine the target data, target node, and request type of the aggregation access request.

[0031] The access module is further configured to route the aggregated access request to the corresponding target HDFS cluster in the aggregated cluster based on the target data, target node, and request type of the aggregated access request, and to access the data blocks in the target HDFS cluster.

[0032] Furthermore, to achieve the above objectives, this application also proposes a distributed cluster aggregation access device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the distributed cluster aggregation access method as described above.

[0033] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the aggregated access method for distributed clusters as described above.

[0034] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the aggregated access method for distributed clusters as described above.

[0035] This application provides a method for aggregated access to distributed clusters. Based on a unified namespace, it aggregates multiple distributed HDFS clusters to form an aggregated cluster. It then acquires aggregated access requests from HDFS clients to the aggregated cluster, parses these requests, and determines the target data, target node, and request type. Based on these parameters, the aggregated access requests are routed to the corresponding target HDFS cluster within the aggregated cluster to access data blocks within that cluster. This application utilizes a unified namespace to aggregate widely distributed HDFS clusters, forming a concise and unified data access view. This allows HDFS clients to easily and uniformly access data in each cluster. Through intelligent request analysis and routing mechanisms, it parses the content of aggregated requests in real time and routes requests to appropriate HDFS clusters based on the actual physical location of the data and the needs of the HDFS client. This ensures seamless and transparent data access, enabling data interaction and solving the technical problem of traditional methods failing to achieve unified access management for multiple widely distributed HDFS clusters. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating an embodiment of the distributed cluster aggregation access method of this application;

[0039] Figure 2 A schematic diagram of the unified namespace for the aggregation access method of the distributed cluster provided in Embodiment 1 of this application;

[0040] Figure 3 This is a schematic diagram of the overall architecture of the distributed cluster aggregation access method provided in Embodiment 1 of this application;

[0041] Figure 4 The address mapping representation of the aggregation access method for a distributed cluster provided in Embodiment 1 of this application;

[0042] Figure 5 This is a flowchart illustrating Embodiment 2 of the distributed cluster aggregation access method of this application;

[0043] Figure 6 This is a flowchart illustrating Embodiment 3 of the distributed cluster aggregation access method of this application;

[0044] Figure 7 This is a schematic diagram of the module structure of the aggregation access device for a distributed cluster according to an embodiment of this application;

[0045] Figure 8 This is a schematic diagram of the hardware operating environment involved in the aggregation access method of the distributed cluster in the embodiments of this application.

[0046] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0047] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0048] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0049] The main solution of this application embodiment is as follows: based on a unified namespace, multiple scattered HDFS clusters are aggregated to obtain an aggregated cluster; the aggregated access request from the HDFS client to the aggregated cluster is obtained, the aggregated access request is parsed, and the target data, target node, and request type of the aggregated access request are determined; based on the target data, target node, and request type of the aggregated access request, the aggregated access request is routed to the corresponding target HDFS cluster in the aggregated cluster to access the data blocks in the target HDFS cluster.

[0050] Currently, access to HDFS clusters typically employs a combination of the HDFS federation mechanism and the ViewFS mechanism. This significantly increases the scalability and fault tolerance of a single HDFS cluster and provides a simple, unified access view for HDFS clients. However, the ViewFS mechanism requires configuration on all clients, making management complex. Different clients require different configurations, and adding a new client or user directory necessitates a complete configuration overhaul, increasing the management burden and complexity of later maintenance. Furthermore, it still focuses on a single HDFS cluster and cannot achieve unified access management for multiple widely distributed HDFS clusters.

[0051] This application provides a solution that aggregates widely distributed HDFS clusters using a unified namespace to form a concise and unified data access view. This enables HDFS clients to easily and uniformly access data in each cluster. Through intelligent request analysis and routing mechanisms, the content of aggregated requests is parsed in real time, and requests are routed to the appropriate HDFS cluster based on the actual physical location of the data and the needs of the HDFS client. This ensures seamless and transparent data access and enables data interaction, solving the technical problem of traditional methods being unable to achieve unified access management for multiple widely distributed HDFS clusters.

[0052] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, a distributed cluster aggregation access device, etc. This embodiment does not specifically limit it. The following uses a distributed cluster aggregation access device as an example to describe this embodiment and the following embodiments.

[0053] This application provides an aggregated access method for a distributed cluster, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the aggregation access method for distributed clusters in this application.

[0054] In this embodiment, the aggregation access method for the distributed cluster includes steps S10 to S30:

[0055] Step S10: Based on a unified namespace, aggregate multiple scattered HDFS clusters to obtain an aggregated cluster;

[0056] It should be noted that the aggregated cluster is the aggregated HDFS cluster. On top of the widely distributed HDFS clusters, a globally unified namespace is established through namespace mapping, realizing the aggregation of access to the distributed HDFS clusters and forming a unified data access view.

[0057] In one feasible implementation, the step of constructing a unified namespace may include steps S101 to S102:

[0058] Step S101: Obtain the file space of the HDFS cluster. The file space contains a file directory tree, and the file directory trees of different HDFS clusters are independent of each other.

[0059] It should be noted that each HDFS cluster's file space consists of a file directory tree, and the file directory trees of different HDFS clusters are independent of each other. For example, refer to... Figure 2 The diagram shows three HDFS clusters, each with its own independent file directory tree. hdfs1 is the root directory of HDFS cluster 1, hdfs2 is the root directory of HDFS cluster 2, and hdfs3 is the root directory of HDFS cluster 3. The root directory can contain different directories such as home, bin, and usr, and the home, bin, and usr directories can also contain subdirectories.

[0060] Step S102: Add a file directory mapping above the root directory of the file directory tree to obtain the unified namespace of the HDFS cluster.

[0061] It is understandable that by adding a layer of file directory mapping above the root directory, the aggregation and construction of a unified namespace can be completed.

[0062] It should be understood that, to achieve a globally unified namespace, a combination of flat and hierarchical directory structures is used. This allows for unified namespace management of the lower-level, widely distributed HDFS clusters, while the upper layer employs a single management node to manage the entire namespace, thus masking the geographical differences between different HDFS clusters. (Reference) Figure 3 In this embodiment, the unified namespace is configured with an HDFS access aggregation management node and an HDFS access aggregation client.

[0063] It should be noted that the HDFS access aggregation management node has at least the functions of cluster registration and management, namespace mapping management, and logging. Regarding cluster registration and management, the HDFS access aggregation management node can register and remove clusters. Through the API (Application Programming Interface), it can register HDFS clusters that need management to the management node or remove clusters that are no longer needed. The node can also monitor cluster status, continuously obtaining the status of each HDFS cluster and confirming the health of the cluster by sending periodic heartbeat checks. When an anomaly occurs in a cluster, the management node will trigger an alarm mechanism and promptly notify the administrator. Regarding namespace mapping management, the HDFS access aggregation management node can achieve geographical transparency. HDFS clients only need to access data in any HDFS cluster through the aggregated logical path. The node can also perform mapping backup and recovery. Considering system robustness and data security, the management node will periodically back up namespace mapping information to ensure rapid recovery in the event of a failure. Regarding the logging function, the HDFS access aggregation management node can record all requests, system activities, and abnormal events in detail, which facilitates administrators in fault location, system performance evaluation, and security analysis, ensuring traceability and transparency.

[0064] Additionally, it should be noted that the HDFS access aggregation client at least has functions such as receiving aggregation access requests, request filtering and defense, and traffic analysis and limiting. Regarding the function of receiving aggregation access requests, the HDFS access aggregation client acts as middleware between the HDFS client and the HDFS access aggregation management node, providing a unified and standardized HDFS access interface, shielding the upper-layer HDFS client from the complexity of the HDFS access aggregation system. For the function of request filtering and defense, the HDFS access aggregation client can perform basic request filtering, such as determining whether the logical address in the aggregation request meets system requirements and whether the prefix exists in the address mapping table, to prevent malicious or non-compliant requests from entering the system, thus improving system security. Regarding the function of traffic analysis and limiting, the HDFS access aggregation client can implement traffic limiting policies for abnormal request traffic (such as sudden traffic spikes or a large number of requests from a single source) to protect the management node from overload.

[0065] In the specific implementation, refer to Figure 4The HDFS access aggregation management node maintains an address mapping table. This table records specific directory mappings. For example, if the NameNode access entry address for HDFS cluster 1 is 10.212.67.8, its mapped path prefix is ​​ / hdfs1. This means that if a user wants to access a file named 1.txt located in the root directory / of HDFS cluster 1, they need to access it through the mapped aggregation access path / hdfs1 / 1.txt. The address mapping table is dynamically maintained based on the registration and deletion of HDFS clusters. To include an HDFS cluster in a unified namespace for centralized management, it only needs to be registered with the HDFS access aggregation management node. The HDFS access aggregation management node will then manage it centrally, perform periodic health checks, and monitor the cluster's operational status in real time.

[0066] Understandably, when registering an HDFS cluster with the HDFS access aggregation management node, the HDFS access aggregation management node can specify a path mapping prefix for the HDFS cluster and accelerate the mapping and conversion speed of logical addresses through flattened metadata management. The mapping relationship is stored in the form of (K,V) data pairs, where K is the address of the NameNode node in the HDFS cluster, and V is the path prefix after address mapping. For example, the physical address of cluster 10.212.67.8 is mapped to the path / beijing, and the physical address and department of cluster 10.212.67.12 are mapped to the path / wuhan / team1. At this time, for the HDFS client, if it wants to access the file located at the physical address / a.txt in 10.212.67.8, the corresponding logical address is / beijing / a.txt, and if it wants to access the file located at the physical address / b.txt in 10.212.67.12, the corresponding logical address is / wuhan / team1 / b.txt.

[0067] Step S20: Obtain the aggregation access request from the HDFS client to the aggregation cluster, parse the aggregation access request, and determine the target data, target node, and request type of the aggregation access request;

[0068] It should be noted that the core architecture of an HDFS cluster consists of a NameNode node responsible for managing metadata and multiple DataNode nodes responsible for data storage. HDFS clients interact with these two types of nodes to meet their data storage and retrieval needs. Therefore, to correctly route HDFS client aggregation requests to the specified NameNode / DataNode nodes in the HDFS cluster, this embodiment introduces a request analysis and routing mechanism. By deeply analyzing the client's aggregation requests, it ensures that each request is accurately sent to the correct target node.

[0069] Additionally, it's important to note that aggregate access requests are requests to access data within an aggregate cluster. HDFS client access requests are typically sent directly to the HDFS aggregate access client, which filters them. Therefore, the aggregate access requests obtained have usually already been filtered by the HDFS aggregate access client, and only those that conform to the specifications are retained. The target node of an aggregate access request is the node that the aggregate access request needs to be routed to; this can be either a NameNode (metadata management node) or a DataNode (data storage node). The target data of an aggregate access request is the data that the aggregate access request wants to access, i.e., the corresponding data in the target node. Aggregate access requests include two types: read requests and write requests. When the request type is read, only the target data is read; when the request type is write, data is usually also written (added data).

[0070] Understandably, after an HDFS client submits an aggregation access request, the HDFS aggregation client filters all such requests. For example, it checks if the logical address in the aggregation request meets system requirements and if the prefix exists in the address mapping table, filtering out malicious or non-compliant requests to improve security. Then, it parses these compliant aggregation access requests to obtain the corresponding target data, target node, and request type. This allows the aggregation access requests to be routed to the correct nodes in the HDFS cluster, enabling access and manipulation of the data on those nodes.

[0071] Step S30: Based on the target data, target node, and request type of the aggregated access request, the aggregated access request is routed to the corresponding target HDFS cluster in the aggregated cluster to access the data blocks in the target HDFS cluster.

[0072] It should be noted that the target HDFS cluster is the HDFS cluster to which the aggregate access request needs to be routed. The target data that the aggregate access request wants to access is usually located in the nodes of the target HDFS cluster.

[0073] It is understood that in this embodiment, data is not simply stored as files, but is broken down into multiple "blocks" for storage. A huge file can be subdivided into multiple data blocks and distributed throughout the HDFS cluster. This block-based data organization method gives HDFS excellent scalability and data redundancy, thereby improving the robustness and fault tolerance of the system, accelerating data reading efficiency, reducing the average data access latency, and improving the overall throughput of the system.

[0074] It should be understood that, in order to further accelerate data access speed, a distributed caching mechanism is adopted. All requests to the NameNode are routed to the HDFS access aggregation management node for processing. Of all access requests to the DataNode, only read requests that hit the cache are routed to the HDFS access aggregation management node for processing. Read requests that miss the cache and all write requests are directly routed to the target HDFS cluster.

[0075] This embodiment provides a method for aggregated access to distributed clusters. Based on a unified namespace, multiple distributed HDFS clusters are aggregated to form an aggregated cluster. Aggregated access requests from HDFS clients to the aggregated cluster are obtained, parsed, and the target data, target node, and request type are determined. Based on the target data, target node, and request type, the aggregated access request is routed to the corresponding target HDFS cluster within the aggregated cluster to access data blocks in the target HDFS cluster. By using a unified namespace to aggregate widely distributed HDFS clusters, a concise and unified data access view is formed, enabling HDFS clients to easily and uniformly access data in each cluster. Through intelligent request analysis and routing mechanisms, the content of aggregated requests is parsed in real time, and requests are routed to appropriate HDFS clusters based on the actual physical location of the data and the needs of the HDFS client, ensuring seamless and transparent data access and enabling data interaction.

[0076] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 Step S30 may include steps S301 to S304:

[0077] Step S301: When the target node of the aggregation access request is a data storage node, obtain the request return information of the metadata management node in the aggregation cluster;

[0078] It's important to note that before accessing a file, every read / write request must send a `getBlocksLocation` request to the NameNode. The request returns an array, where each element stores the data block corresponding to that file and the physical address of the DataNode containing that data block. The returned information is the result of the request from the NameNode, containing relevant information about the target data block.

[0079] In this embodiment, if the target node of the aggregation access request is a data storage node, the request return information of the metadata management node in the aggregation cluster is obtained for subsequent analysis.

[0080] Step S302: Based on the request return information, determine the target data block of the target data of the aggregated access request in the target node and the physical address of the target node corresponding to the target data block;

[0081] It is understandable that the target data block in the target node is the same as the target data block containing the target data, and the physical address of the target node corresponding to the target data block is the same as the physical address of the target node containing the target data block. By intercepting the request results returned by the NameNode, and analyzing them, the elements in the array are parsed one by one to extract the relevant information of the target data block, such as: the target data block containing the target data, and the physical address of the target node containing the target data block.

[0082] Step S303: When the request type of the aggregated access request is a read request, obtain the cached data block of the aggregated cluster;

[0083] It's important to note that cached data blocks are frequently accessed data blocks that are cached. In this embodiment, when the aggregated access request type is a read request, the cached data blocks of the aggregation cluster are retrieved. Instead of caching the entire file, hot data is identified and captured on a block-by-block basis, and then frequently accessed data blocks are cached. This allows for faster location and access to hot data, especially for large files, significantly reducing unnecessary data loading and processing time. The usage frequency and access patterns of each data block are monitored so that the system can better determine which data should be cached. This caching strategy not only improves data access speed but also reduces network bandwidth and storage resource consumption.

[0084] Step S304: When the target data block corresponding to the aggregated access request matches the cached data block, the aggregated access request is determined to be a cache hit read request. Based on the cache address of the cached data block, the cache hit read request is routed to the corresponding cached data block and the cached data block is accessed.

[0085] It should be noted that the existence of the target data block in the cached data block can be determined by the unique identifier (blockid) of the target data block. If the target data block matches the cached data block, that is, the target data block exists in the cached data block, the aggregated access request is considered a cache hit read request. An additional cache address (i.e., the address of the local cache) is inserted into the specified position of the request return information. The cache hit read request can access the target data in the cached data block.

[0086] In one feasible implementation, when the target data block corresponding to the aggregated access request does not match the cached data block, the aggregated access request is determined to be a non-cache hit read request. Based on the physical address of the target node, the target HDFS cluster of the cache hit read request is determined, and the non-cache hit read request is routed to the corresponding data storage node in the target HDFS cluster to access the target data block.

[0087] Understandably, if the target data block does not match the cached data block, meaning the target data block does not exist in the cached data block, the aggregation access request is considered a non-cache hit read request. Therefore, no measures are taken, and the non-cache hit read request is directly routed to the corresponding data storage node in the target HDFS cluster. This ensures that only cache hit read requests are routed to the HDFS access aggregation management node for processing, while non-cache hit read requests are directly routed to the DataNode node in the target HDFS cluster for data access and operation.

[0088] In one feasible implementation, when the request type of the aggregated access request is a write request, the target HDFS cluster of the aggregated access request is determined based on the physical address of the target node, and the aggregated access request is routed to the corresponding data storage node in the target HDFS cluster to access the target data block.

[0089] Understandably, write requests to DataNode nodes are directly routed to the DataNode nodes in the target HDFS cluster for data access and manipulation.

[0090] In one feasible implementation, the step of determining cached data blocks may include: obtaining the status of data blocks of data storage nodes in the aggregation cluster; selecting data blocks in the fully written state as candidate data blocks; and determining the cached data blocks of the aggregation cluster based on the data popularity of the candidate data blocks.

[0091] It should be noted that HDFS only supports append-only data writes, which means that once data is written, it will remain unchanged unless it is explicitly deleted. Therefore, you can safely cache data blocks without worrying about data inconsistency issues.

[0092] Additionally, it's important to note that when an HDFS client prepares to read a data block, it first sends a `getVisibleLength` request to the DataNode. This request aims to obtain the actual size of the data block on the physical storage medium, providing an accurate reference for subsequent data reading. Using this strategy, this embodiment can accurately determine the status of a data block, particularly whether it's full, by sending a simulated request to the DataNode. If a data block has been completely written, meaning no further data will be added, it becomes an ideal candidate for a fine-grained distributed caching strategy. Candidate data blocks are those that are likely to be cached; cached data blocks can be selected from among these candidate blocks.

[0093] Understandably, this embodiment employs a fine-grained distributed caching strategy based on file blocks. Data blocks in a fully written state are considered candidate data blocks. Based on the data popularity (frequency of access to data within a block) of these candidate blocks, the most popular data blocks are cached. This allows for more flexible responses to changes in data popularity; when the access frequency of a data block decreases, it can be automatically removed from the cache, freeing up space for other more frequently accessed data blocks. This dynamic adjustment ensures optimal cache utilization, thereby achieving more efficient data management and improved overall performance. This is of great significance for processing large datasets, improving the efficiency of data analysis tasks, and reducing data access latency, especially for data-intensive application scenarios.

[0094] In its implementation, by adopting a fine-grained distributed caching strategy based on file blocks, it avoids many of the complexity and inconsistency problems of traditional caching strategies, and ensures the efficiency, stability and accuracy of data access, accelerating the data access process with an average speedup of 40%.

[0095] This embodiment provides an aggregation access method for distributed clusters. It utilizes a unified namespace to aggregate widely distributed HDFS clusters, forming a concise and unified data access view. This allows HDFS clients to easily and uniformly access data in each cluster. Through intelligent request analysis and routing mechanisms, the content of aggregated requests is parsed in real time, and requests are routed to the appropriate HDFS cluster based on the actual physical location of the data and the needs of the HDFS client. This ensures seamless and transparent data access, enables data interaction, and achieves efficient data access based on a fine-grained distributed caching strategy using file blocks.

[0096] Based on the first embodiment of this application, in the third embodiment of this application, the same or similar content as the above embodiment can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 6Step S30 may include steps S301' to S302':

[0097] Step S301': When the target node of the aggregated access request is a metadata management node, determine the physical address of the target HDFS cluster corresponding to the target data based on the logical address of the target data of the aggregated access request, and adjust the aggregated access request accordingly.

[0098] Understandably, for requests to the NameNode, the aggregated access request is analyzed, the logical address is parsed according to predetermined rules, the address mapping table is consulted to obtain the mapping relationship, and the logical address is converted into the corresponding physical address. Afterwards, the HDFS access aggregation management node modifies the request body appropriately and then routes it to the NameNode in the target HDFS cluster.

[0099] In one feasible implementation, the step of determining the physical address of the target HDFS cluster corresponding to the target data based on the logical address of the target data may include: obtaining the address mapping table of the aggregated cluster, wherein the address mapping table stores the physical address of the HDFS cluster and the logical address mapped from the physical address of the HDFS cluster; determining the physical address matching the logical address of the target data in the address mapping table based on the logical address of the target data; taking the HDFS cluster corresponding to the matching physical address as the target HDFS cluster corresponding to the target data, and obtaining the physical address of the target HDFS cluster.

[0100] It should be noted that the reference Figure 4 The address mapping table stores the physical addresses of the HDFS cluster and the logical addresses mapped from those physical addresses. Based on the logical address of the target data, a matching physical address is found in the address mapping table. Thus, the HDFS cluster with the matching physical address can be used as the target HDFS cluster, and the matching physical address can be used as the physical address of the target HDFS cluster. This enables the aggregation access request to be routed to the NameNode node in the target HDFS cluster, ensuring the efficiency, stability, and accuracy of data access.

[0101] Step S302': Based on the physical address of the target HDFS cluster, the adjusted aggregated access request is routed to the metadata management node of the corresponding target HDFS cluster to access the data blocks in the metadata management node of the target HDFS cluster.

[0102] In its implementation, by employing a unified namespace technology, HDFS clients can send various aggregated access requests to the HDFS access aggregation client, thus routing all requests targeting the NameNode to the local system. Upon receiving these requests, the HDFS access aggregation client, based on predefined filtering rules, forwards eligible aggregated access requests to the HDFS access aggregation management node. This node analyzes the request, resolves the logical address according to predetermined rules, and looks up the mapping relationship in the address mapping table, converting the logical address to a physical address. The HDFS access aggregation management node then modifies the request body appropriately and routes it to the specified NameNode node in the HDFS cluster.

[0103] This embodiment provides an aggregation access method for distributed clusters. It uses a unified namespace to aggregate widely distributed HDFS clusters, forming a concise and unified data access view. This allows HDFS clients to easily and uniformly access data in each cluster. Through intelligent request analysis and routing mechanisms, the content of aggregated requests is parsed in real time, and requests are routed to the appropriate HDFS cluster based on the actual physical location of the data and the needs of the HDFS client, ensuring seamless and transparent data access. Furthermore, it achieves efficient data access based on a fine-grained distributed caching strategy using file blocks.

[0104] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the aggregation access method of the distributed cluster in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0105] This application also provides an aggregation access device for distributed clusters, please refer to... Figure 7 The aggregated access device for the distributed cluster includes:

[0106] Aggregation module 10 is used to aggregate multiple scattered HDFS clusters based on a unified namespace to obtain an aggregated cluster;

[0107] Access module 20 is used to obtain the aggregation access request of the HDFS client to the aggregation cluster, parse the aggregation access request, and determine the target data, target node and request type of the aggregation access request;

[0108] The access module 20 is further configured to route the aggregated access request to the corresponding target HDFS cluster in the aggregated cluster based on the target data, target node, and request type of the aggregated access request, and access the data blocks in the target HDFS cluster.

[0109] In one feasible implementation, the access module 20 is further configured to, when the target node of the aggregated access request is a metadata management node, determine the physical address of the target HDFS cluster corresponding to the target data based on the logical address of the target data of the aggregated access request, and adjust the aggregated access request accordingly.

[0110] Based on the physical address of the target HDFS cluster, the adjusted aggregated access request is routed to the metadata management node of the corresponding target HDFS cluster to access and operate on the data blocks in the metadata management node of the target HDFS cluster.

[0111] In one feasible implementation, the access module 20 is further configured to obtain the address mapping table of the aggregation cluster, the address mapping table storing the physical address of the HDFS cluster and the logical address mapped from the physical address of the HDFS cluster.

[0112] Based on the logical address of the target data, determine the physical address that matches the logical address of the target data in the address mapping table;

[0113] The HDFS cluster corresponding to the matched physical address is taken as the target HDFS cluster corresponding to the target data, and the physical address of the target HDFS cluster is obtained.

[0114] In one feasible implementation, the access module 20 is further configured to obtain request return information from the metadata management node in the aggregation cluster when the target node of the aggregation access request is a data storage node;

[0115] Based on the request return information, determine the target data block of the target data of the aggregated access request in the target node and the physical address of the target node corresponding to the target data block;

[0116] When the request type of the aggregated access request is a write request, the target HDFS cluster of the aggregated access request is determined based on the physical address of the target node, and the aggregated access request is routed to the corresponding data storage node in the target HDFS cluster to access the target data block.

[0117] In one feasible implementation, the access module 20 is further configured to obtain the cached data block of the aggregation cluster when the request type of the aggregation access request is a read request;

[0118] When the target data block corresponding to the aggregated access request matches the cached data block, the aggregated access request is determined to be a cache hit read request. Based on the cache address of the cached data block, the cache hit read request is routed to the corresponding cached data block and the cached data block is accessed.

[0119] When the target data block corresponding to the aggregated access request does not match the cached data block, the aggregated access request is determined to be a non-cache hit read request. Based on the physical address of the target node, the target HDFS cluster of the cache hit read request is determined, and the non-cache hit read request is routed to the corresponding data storage node in the target HDFS cluster to access the target data block.

[0120] In one feasible implementation, the access module 20 is further configured to obtain the status of data blocks of data storage nodes in the aggregation cluster;

[0121] Data blocks in the fully written state are selected as candidate data blocks;

[0122] Based on the data popularity of the candidate data blocks, the cached data blocks of the aggregation cluster are determined.

[0123] The distributed cluster aggregation access device provided in this application, employing the distributed cluster aggregation access method in the above embodiments, addresses the technical problem of unified access management for multiple widely distributed HDFS clusters, which is difficult to achieve using traditional methods. Compared with the prior art, the beneficial effects of the distributed cluster aggregation access device provided in this application are the same as those of the distributed cluster aggregation access method provided in the above embodiments, and other technical features in the distributed cluster aggregation access device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0124] This application provides a distributed cluster aggregation access device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the distributed cluster aggregation access method in Embodiment 1 above.

[0125] The following is for reference. Figure 8 This document illustrates a schematic diagram of a distributed cluster aggregation access device suitable for implementing embodiments of this application. The distributed cluster aggregation access device in this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8The aggregated access device for the distributed cluster shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0126] like Figure 8 As shown, the aggregated access device of the distributed cluster may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the aggregated access device of the distributed cluster. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the aggregated access devices of a distributed cluster to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows aggregated access devices of a distributed cluster with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented or possessed alternatively.

[0127] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0128] The distributed cluster aggregation access device provided in this application, employing the distributed cluster aggregation access method in the above embodiments, can solve the technical problem that traditional methods struggle to achieve unified access management for multiple widely distributed HDFS clusters. Compared with the prior art, the beneficial effects of the distributed cluster aggregation access device provided in this application are the same as those of the distributed cluster aggregation access method provided in the above embodiments, and other technical features in this distributed cluster aggregation access device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0129] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0130] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0131] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the aggregated access method for the distributed cluster in the above embodiments.

[0132] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0133] The aforementioned computer-readable storage medium may be included in the aggregated access device of the distributed cluster; or it may exist independently and not be assembled into the aggregated access device of the distributed cluster.

[0134] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the aggregation access device of the distributed cluster, the aggregation access device of the distributed cluster: obtains the aggregation access request from the HDFS client to the aggregation cluster, parses the aggregation access request, determines the target data, target node, and request type of the aggregation access request; and, based on the target data, target node, and request type of the aggregation access request, routes the aggregation access request to the corresponding target HDFS cluster in the aggregation cluster, and accesses the data blocks in the target HDFS cluster.

[0135] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0137] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0138] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described aggregated access method for distributed clusters. This solves the technical problem that traditional methods struggle to achieve unified access management for multiple widely distributed HDFS clusters. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the aggregated access method for distributed clusters provided in the above embodiments, and will not be elaborated upon here.

[0139] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described distributed cluster aggregation access method.

[0140] The computer program product provided in this application can solve the technical problem that traditional methods are difficult to implement unified access management for multiple widely distributed HDFS clusters. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the aggregated access method for distributed clusters provided in the above embodiments, and will not be repeated here.

[0141] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for aggregated access to a dispersed cluster, characterized by, The method comprises: Aggregating a plurality of HDFS clusters based on a uniform namespace to obtain an aggregated cluster; Obtaining an aggregated access request of an HDFS client to the aggregated cluster, parsing the aggregated access request, and determining target data, a target node, and a request type of the aggregated access request; Based on the target data, the target node, and the request type of the aggregated access request, routing the aggregated access request to a corresponding target HDFS cluster in the aggregated cluster, and accessing data blocks in the target HDFS cluster, specifically comprising: When the target node of the aggregated access request is a metadata management node, determining a physical address of the target HDFS cluster corresponding to the target data based on a logical address of the target data of the aggregated access request, adjusting the aggregated access request, routing the adjusted aggregated access request to the metadata management node of the corresponding target HDFS cluster based on the physical address of the target HDFS cluster, and accessing data blocks in the metadata management node of the target HDFS cluster; When the target node of the aggregated access request is a data storage node, obtaining request return information of a metadata management node in the aggregated cluster, determining a target data block of the target data in the target node and a physical address of the target node corresponding to the target data block based on the request return information; When the request type of the aggregated access request is a write request, determining the target HDFS cluster of the aggregated access request based on the physical address of the target node, and routing the aggregated access request to the corresponding data storage node in the target HDFS cluster to access the target data block; When the request type of the aggregated access request is a read request, obtaining cache data blocks of the aggregated cluster; when the target data block corresponding to the aggregated access request meets the cache data blocks, determining that the aggregated access request is a cache hit read request, routing the cache hit read request to the corresponding cache data block based on a cache address of the cache data block, and accessing the cache data block; when the target data block corresponding to the aggregated access request does not meet the cache data blocks, determining that the aggregated access request is a non-cache hit read request, determining the target HDFS cluster of the cache hit read request based on the physical address of the target node, and routing the non-cache hit read request to the corresponding data storage node in the target HDFS cluster to access the target data block; The method further comprises: obtaining the state of the data blocks of the data storage nodes in the aggregated cluster, taking data blocks in a completely written state as candidate data blocks, and determining cache data blocks of the aggregated cluster based on the data hotness of the candidate data blocks, wherein if there are data blocks with decreased data hotness in the cache data blocks, the data blocks with decreased data hotness are removed to re-determine data blocks with higher data hotness as cache data blocks.

2. The method of claim 1, wherein, The step of determining the physical address of the target HDFS cluster corresponding to the target data based on the logical address of the target data comprises: obtaining an address mapping table of the aggregated cluster, the address mapping table storing the physical address of the HDFS cluster and the logical address of the HDFS cluster after the physical address is mapped; determining the physical address matched with the logical address of the target data in the address mapping table based on the logical address of the target data; taking the HDFS cluster corresponding to the matched physical address as the target HDFS cluster corresponding to the target data, and obtaining the physical address of the target HDFS cluster.

3. An aggregated access device for a dispersed cluster, characterized by, The device comprises: an aggregation module configured to aggregate a plurality of HDFS clusters based on a unified namespace to obtain an aggregated cluster; an access module configured to obtain an aggregated access request of an HDFS client to the aggregated cluster, analyze the aggregated access request, and determine target data, a target node, and a request type of the aggregated access request; the access module is further configured to route the aggregated access request to a corresponding target HDFS cluster in the aggregated cluster based on the target data, the target node, and the request type of the aggregated access request, and access a data block in the target HDFS cluster; the access module is further configured to, when the target node of the aggregated access request is a data storage node, obtain request return information of a metadata management node in the aggregated cluster, determine a target data block of the target data in the target node and a physical address of a target node corresponding to the target data block based on the request return information; when the request type of the aggregated access request is a write request, determine the target HDFS cluster of the aggregated access request based on the physical address of the target node, and route the aggregated access request to a corresponding data storage node in the target HDFS cluster to access the target data block; the access module is further configured to, when the request type of the aggregated access request is a read request, obtain a cache data block of the aggregated cluster; when the target data block corresponding to the aggregated access request matches the cache data block, determine that the aggregated access request is a cache hit read request, route the cache hit read request to the corresponding cache data block based on a cache address of the cache data block, and access the cache data block; when the target data block corresponding to the aggregated access request does not match the cache data block, determine that the aggregated access request is a non-cache hit read request, determine the target HDFS cluster of the cache hit read request based on the physical address of the target node, and route the non-cache hit read request to a corresponding data storage node in the target HDFS cluster to access the target data block; the access module is further configured to, when the target node of the aggregated access request is a metadata management node, determine the physical address of the target HDFS cluster corresponding to the target data based on the logical address of the target data of the aggregated access request, and adjust the aggregated access request; Based on the physical address of the target HDFS cluster, the adjusted aggregated access request is routed to a metadata management node of the corresponding target HDFS cluster for accessing data blocks in the metadata management node of the target HDFS cluster; The access module is further configured to acquire the state of the data blocks of the data storage nodes in the aggregated cluster; The data block in the completely written state is taken as a candidate data block; Based on the data hotness of the candidate data block, the cached data block of the aggregated cluster is determined, wherein if there is a data block with decreased data hotness in the cached data block, the data block with decreased data hotness is removed, and the data block with higher data hotness is determined as the cached data block.

4. A disaggregated cluster of converged access devices, comprising: The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the aggregated access method of the dispersed cluster as claimed in claim 1 or 2.

5. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the aggregated access method of the dispersed cluster as claimed in claim 1 or 2.

6. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is executed by the processor to implement the steps of the aggregated access method of the dispersed cluster as claimed in claim 1 or 2.

Images