Data access method and device, electronic equipment and storage medium

By using cache routing tables and access policies in multi-availability zone scenarios, the complexity problem caused by different access rules of the cache service cluster is solved, unified data access across the Availability Zone is achieved, and access efficiency and universality are improved.

CN119003585APending Publication Date: 2024-11-22BEIJING QIYI CENTURY SCI & TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202410992074.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In the prior art, the access rules of the cache service cluster in the multi-availability zone scenario are different, resulting in the complex access process of direct use of services and the lack of a standard cache routing mechanism.

Method used

Provide a data access method, by receiving resource access requests carrying target metadata, using cache routing table to determine the mount status of the target cache resource, and select applicable access policies based on the mount status, and convert them into cache services or underlying files. System access requests to achieve unified access across the Availability Zone.

Benefits of technology

The data access process is simplified, the universality and efficiency of data access is improved, the complexity caused by the differences in different availability zones is avoided, and convenient data access across availability zones is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003585A_ABST
    Figure CN119003585A_ABST
Patent Text Reader

Abstract

The invention provides a data access method and device, electronic equipment and a storage medium, which are applied to the technical field of computers, and the method comprises the following steps: receiving a resource access request carrying target metadata; based on a cache routing table, determining a mounting condition of a target cache resource corresponding to the target metadata, the cache routing table comprising a corresponding relationship between metadata corresponding to the cache resources in different available areas and a mounting path; and accessing source data corresponding to the target cache resource or the target metadata based on an access strategy matched with the mounting condition. Therefore, the difference of different available areas does not need to be paid attention to in the process of accessing the cache resources, only the metadata of the data needing to be accessed needs to be specified, the data access process can be conveniently realized through a set of uniform access strategies, the data access universality is improved, the data access process is simplified, and the data access efficiency is improved. And the data access efficiency is improved to a certain extent.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a data access method, device, electronic device and storage medium. Background Art

[0002] In big data scenarios, there are some hot data that will be accessed repeatedly in large quantities, such as basic result data, key dimension tables, training sample data, etc. Caching these hot data can bring great benefits. 1) Caching them from low-performance storage media (HDD) to high-performance storage media (SSD) improves disk access performance; 2) Caching them from remote storage to nearby computing storage reduces dedicated line network overhead.

[0003] There are already some big data caching solutions in the industry, such as open source Alluxio, Alibaba Cloud Jindofs, Baidu Cloud Rapidfs, etc. These solutions provide the ability to deploy cache services and cache in single availability zone (AZ) scenarios. However, in the cross-AZ storage scenario of big data, cache resources will be deployed in multiple AZs and multiple clusters, but this involves issues such as caching mechanisms and data access consistency in multi-AZ scenarios. The industry currently does not have a standard and universal cache routing mechanism. There are often problems such as different access rules for different cache service clusters, which makes the direct access process of businesses complex. Summary of the invention

[0004] In view of this, the present application provides a data access method, device, electronic device and storage medium to solve the problem in the prior art that different cache service clusters have different access rules and the business directly uses a complex access process.

[0005] A first aspect of the present application provides a data access method, the method comprising:

[0006] receiving a resource access request carrying target metadata;

[0007] Based on the cache routing table, determining the mounting status of the target cache resource corresponding to the target metadata; the cache routing table includes: the corresponding relationship between the metadata corresponding to the cache resources in different available zones and the mounting path;

[0008] The target cache resource or the source data corresponding to the target metadata is accessed based on an access policy matching the mounting situation.

[0009] Optionally, the accessing the target cache resource or the source data corresponding to the target metadata based on the access policy matching the mounting situation includes:

[0010] When the mount condition is that the target mount path corresponding to the target cache resource exists in the cache routing table, determining that the access policy matching the mount condition is the first access policy, and converting the resource access request into an access request to the cache service;

[0011] In response to the access request to the cache service, access the target cache resource based on the first access policy; the first access policy is to access the target cache resource in the available zone cache based on the queried target mount path;

[0012] When the mount situation is that the target mount path corresponding to the target cache resource does not exist in the cache routing table, determine that the access policy matching the mount situation is a second access policy, and convert the resource access request into an access request to the underlying file system;

[0013] In response to the access request to the underlying file system, the source data corresponding to the target metadata is accessed based on the second access policy; the second access policy is to access the source data corresponding to the target metadata in the underlying file system.

[0014] Optionally, when the access operation corresponding to the resource access request is a read operation, accessing the target cache resource in the available zone cache based on the queried target mount path includes:

[0015] Obtaining the first available zone where the source data corresponding to the target metadata in the underlying file system is located, the second available zone set where the target cache resource is located, and the third available zone where the client is located;

[0016] In a case where the second available zone set includes the third available zone, determining a first cache service node from cache service nodes corresponding to the third available zone based on the target mount path and a consistent hashing algorithm;

[0017] In a case where the second availability zone set does not include the third availability zone but includes the first availability zone, determining a first cache service node from cache service nodes corresponding to the first availability zone based on the target mount path and a consistent hashing algorithm;

[0018] Based on the first cache service node, performing a read operation on the target cache resource in the cache of the available zone corresponding to the first cache service node;

[0019] The method further comprises:

[0020] In a case where the second available zone cache set does not include the third available zone and the first available zone, a read operation is performed on the source data corresponding to the target metadata in the underlying file system.

[0021] Optionally, when the access operation corresponding to the resource access request is a write operation, accessing the target cache resource in the available zone cache based on the queried target mount path includes:

[0022] In the case where the write mode corresponding to the write operation is not the target write mode, obtaining a first available area where the source data corresponding to the target metadata in the underlying file system is located; the target write mode is the underlying file system write mode;

[0023] Based on the target mount path and the consistent hashing algorithm, determine a second cache service node from cache service nodes corresponding to the first availability zone;

[0024] Based on the second cache service node, in the cache of the available zone corresponding to the second cache service node, a write operation is performed on the target data carried by the resource access request according to the write mode corresponding to the write operation;

[0025] The method further comprises:

[0026] When the write mode corresponding to the write operation is the target write mode, a write operation is performed on the target data carried in the resource access request in the underlying file system.

[0027] Optionally, the method further comprises:

[0028] When the write mode corresponding to the write operation is not the target write mode and the mount path does not exist in the first available zone cache, access failure information is sent for the resource access request.

[0029] Optionally, the method further comprises:

[0030] The cache routing table is obtained from a data cluster across multiple different availability zones according to a target period, wherein the data cluster includes: cache resource groups corresponding to different availability zones, and the cache routing table is generated based on metadata and storage methods of the cache resource groups.

[0031] Optionally, the method further comprises:

[0032] The cache routing table is obtained from any available zone according to the target period. The cache routing table is generated based on the metadata and storage relationship stored in any available zone. The metadata and storage relationship are synchronized between multiple different available zones.

[0033] Optionally, the method further comprises:

[0034] The cache resources are split according to the target dimension, and the obtained cache resources of each dimension are stored in the corresponding cache resource group.

[0035] According to a second aspect of the present application, a data access device is provided, the device comprising:

[0036] A receiving module, used for receiving a resource access request carrying target metadata;

[0037] A query module, used to determine the mounting status of the target cache resource corresponding to the target metadata based on the cache routing table; the cache routing table includes: the corresponding relationship between the metadata corresponding to the cache resources in different available zones and the mounting path;

[0038] An access module is used to access the target cache resource or the source data corresponding to the target metadata based on an access policy matching the mounting situation.

[0039] Optionally, the access module is further used to:

[0040] When the mount condition is that the target mount path corresponding to the target cache resource exists in the cache routing table, determining that the access policy matching the mount condition is the first access policy, and converting the resource access request into an access request to the cache service;

[0041] In response to the access request to the cache service, access the target cache resource based on the first access policy; the first access policy is to access the target cache resource in the available zone cache based on the queried target mount path;

[0042] When the mount situation is that the target mount path corresponding to the target cache resource does not exist in the cache routing table, determine that the access policy matching the mount situation is a second access policy, and convert the resource access request into an access request to the underlying file system;

[0043] In response to the access request to the underlying file system, the source data corresponding to the target metadata is accessed based on the second access policy; the second access policy is to access the source data corresponding to the target metadata in the underlying file system.

[0044] Optionally, when the access operation corresponding to the resource access request is a read operation, the access module is further configured to:

[0045] Obtaining the first available zone where the source data corresponding to the target metadata in the underlying file system is located, the second available zone set where the target cache resource is located, and the third available zone where the client is located;

[0046] In a case where the second available zone set includes the third available zone, determining a first cache service node from cache service nodes corresponding to the third available zone based on the target mount path and a consistent hashing algorithm;

[0047] In a case where the second availability zone set does not include the third availability zone but includes the first availability zone, determining a first cache service node from cache service nodes corresponding to the first availability zone based on the target mount path and a consistent hashing algorithm;

[0048] Based on the first cache service node, performing a read operation on the target cache resource in the cache of the available zone corresponding to the first cache service node;

[0049] In a case where the second available zone cache set does not include the third available zone and the first available zone, a read operation is performed on the source data corresponding to the target metadata in the underlying file system.

[0050] Optionally, when the access operation corresponding to the resource access request is a write operation, the access module is further configured to:

[0051] In the case where the write mode corresponding to the write operation is not the target write mode, obtaining a first available area where the source data corresponding to the target metadata in the underlying file system is located; the target write mode is the underlying file system write mode;

[0052] Based on the target mount path and the consistent hashing algorithm, determine a second cache service node from cache service nodes corresponding to the first availability zone;

[0053] Based on the second cache service node, in the cache of the available zone corresponding to the second cache service node, a write operation is performed on the target data carried by the resource access request according to the write mode corresponding to the write operation;

[0054] When the write mode corresponding to the write operation is the target write mode, a write operation is performed on the target data carried in the resource access request in the underlying file system.

[0055] Optionally, the access module is further used to:

[0056] When the write mode corresponding to the write operation is not the target write mode and there is no cache service node in the cache of the first available zone, access failure information is sent for the resource access request.

[0057] According to a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data access method when executing the computer program.

[0058] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program implements the above-mentioned data access method when executed by a processor.

[0059] With respect to the prior art, this application has the following advantages:

[0060] The present application provides a data access method and device, which caches cache resources in different availability zones. Therefore, when receiving a resource access request sent by a client, the method and device can query the mount path of the cache resource in the cache routing table storing the metadata based on the target metadata carried in the resource access request to know the mount status of the cache resource, and then select an applicable access strategy for different mount statuses to access the cache resource or source data, so that in the process of accessing the cache resource, there is no need to pay attention to the differences between different availability zones, and only the metadata of the required access data needs to be specified. The data access process can be conveniently implemented through a set of unified access strategies, thereby improving the versatility of data access while simplifying the data access process, and improving the efficiency of data access to a certain extent.

[0061] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0063] Figure 1 is a system schematic diagram of a data access method provided by an embodiment of the present application;

[0064] Figure 2 is a step flow chart of a data access method provided by an embodiment of the present application;

[0065] Figure 3 It is a specific schematic diagram of a data access method provided in an embodiment of the present application;

[0066] Figure 4 is a structural block diagram of a data access device provided in an embodiment of the present application;

[0067] Figure 5 It is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0069] Figure 1 It is a system diagram of a data access method provided in an embodiment of the present application, wherein the system comprises: a client 10, a server 20, a cache service 30, and an underlying file system 40.

[0070] The client can be used to provide data query services to users, and the client can encapsulate the underlying file system (UFS, Under File System) client and cache service client. The client is a unified entry for user access, proxying user requests to access UFS, executing multi-AZ cache routing strategies, and converting user access requests for data into access to the underlying file system (UFS, Under File System) or cache service, so that users do not need to pay attention to the available area type and data cluster type of data storage.

[0071] The server is used to manage routing rules, and can provide the client with an interface for adding, deleting, modifying and checking routing rules. The routing rules are used to convert the data access requests input by the user into access to the underlying file system or cache service. The required access strategy. It is worth noting that in the embodiment of the present application, the routing rules are applicable to various types of availability zones and data clusters, thereby assisting the client to achieve a unified access entry for the underlying data system and cache service. At the same time, the server is also used to generate, manage and maintain the cache routing table based on the metadata obtained from the cache service and the mount path of the cache resource corresponding to the metadata, and can maintain the client configuration information corresponding to the client.

[0072] The cache service can be an alluxio service, a jindofs service, a rapidfs service, etc., which can be set according to actual needs and is not limited here. The cache service can be deployed across multiple availability zones, and the availability zone can include an availability zone cache. The availability zone cache caches the cache resources corresponding to the source data in the underlying file system, and the metadata of the cache resources in each availability zone cache is consistent. Exemplarily, a central node can be set between the availability zones of the cache service, and the central node is used to store the metadata information and mounting status of the cache resources in all the available zone caches in the cache service. The metadata information of the cache resources stored in the central node can be used to access the source data corresponding to the target cache resource or the target metadata in combination with the metadata information of the cache resource after confirming the access policy and the cache service node. Among them, the mounting status can include whether the target mount path corresponding to the target cache resource corresponding to the source data exists, and the mounting status can indicate whether the source data is cached in the cache service, that is, whether the target cache resource corresponding to the source data is cached in the cache service. The mounting status can include the existence of the target mount path corresponding to the target cache resource in the cache routing table, or the absence of the target mount path corresponding to the target cache resource in the cache routing table. Moreover, whenever an access operation occurs in the system, the metadata information stored in the central node is synchronously updated based on the operation content of the access operation. In this way, cross-zone access to cache resources can be achieved by sharing the metadata corresponding to cache resources between different availability zones. Furthermore, in order to improve the manageability of cache resources in the cache service, cache resources can be classified and stored according to classification methods such as data type or data access path, that is, multiple cache service clusters are divided in the cache service to store different types of cache resources. For example, cache resources can be split according to different dimensions such as business lines (advertising, search, recommendation), application scenarios (Trino acceleration, data production), data types (HDFS directory, HIVE, ICEBERG), etc., and stored in different cache service clusters.

[0073] The underlying file system may be a system composed of a variety of different underlying file architectures, and the underlying file architecture may be, for example, AWS S3, OSS, and the like.

[0074] It is worth noting that in order to ensure the stability and feasibility of data transmission between different types of availability zones and data clusters, the system protocols used by the client, the underlying file system and the cache service need to be unified. And different types of availability zones and data clusters in the cache service and the underlying file system also need to support the system protocol. The system protocol can be, for example, HDFS (Hadoop Distributed File System). HDFS can provide high-throughput data access and is very suitable for application scenarios where data is accessed on large-scale data sets. Of course, this is only an exemplary description. The system protocol is also a protocol other than HDFS. It can be set according to actual needs and is not limited here.

[0075] Specifically, during the data access process, the user can use the client as a unified entrance for data access and input a resource access request to the client. The client will query the mounting status of the target cache resource to be accessed according to the cache routing table provided by the server, and then query the access policy that matches the mounting status in the cache routing rules provided by the server based on the mounting status. The user's resource access request is converted into an access request for the cache service or the underlying file system based on the access policy. Then, the client will access the cache service or the underlying file through the cache service client or the underlying file system (UFS, Under File System) client based on the converted access request, thereby completing the access process to the cache resources or source data.

[0076] The embodiment of the present application caches cache resources in different availability zones, so that when a resource access request sent by a client is received, the mount path of the cache resource corresponding to the target metadata can be queried in the cache routing table storing the metadata based on the target metadata carried in the resource access request to understand the mount status of the cache resource, and then an applicable access strategy is selected according to different mount situations to access the cache resources in the cache service or the source data in the underlying file system, so that in the process of accessing data, there is no need to pay attention to the differences between different availability zones, and only the metadata of the required access data needs to be specified. The cache resource access process can be conveniently implemented by accessing the cache resource through a set of unified access strategies.

[0077] Figure 2 : is a flowchart of a data access method provided by an embodiment of the present application, the method comprising:

[0078] Step 101: Receive a resource access request carrying target metadata.

[0079] It should be noted that the target metadata refers to various descriptive information of the source data to be accessed, such as the data type, data volume, storage address, storage method, etc. of the source data, which can be set according to actual needs and is not limited here. The types of resource access requests can include resource read requests and resource write requests.

[0080] It is understandable that the cache resources in the cache service are obtained by caching the source data in the underlying file system. Therefore, the source data in the underlying file system and the cache resources corresponding to the source data in the cache service can be determined based on the metadata.

[0081] In the embodiments of the present application, Figure 1 The client can provide a user interface in the form of an application program interface or a web page window. The user instructs the client to generate a resource access request by inputting target metadata corresponding to the cache resource to be accessed in the user interface.

[0082] Step 102: Determine the mounting status of the target cache resource corresponding to the target metadata based on the cache routing table; the cache routing table includes: the correspondence between the metadata corresponding to the cache resources in different availability zones and the mounting paths.

[0083] In an embodiment of the present application, the client periodically obtains and updates the cache routing table from the server, and the cache routing table may include the correspondence between the metadata corresponding to different cache resources and the mount path. The server determines the mount path based on the storage location of the cache resources in each available zone, and then determines the correspondence between the metadata corresponding to the cache resource and the mount path. Thus, the client can compare the target metadata carried in the resource access request with the query cache routing table to determine the mount status of the target cache resource, that is, to determine the target mount path corresponding to the target cache resource corresponding to the required access source data. Of course, it is also possible that the target mount path cannot be queried, which can be used as a reference for the subsequent selection of access strategies. Correspondingly, the mount status may include that the target mount path corresponding to the target cache resource exists in the cache routing table, or that the target mount path corresponding to the target cache resource does not exist in the cache routing table. Exemplarily, the target metadata can be compared one by one with the metadata corresponding to different cache resources in the cache routing table. When there is at least one cache resource whose metadata matches the target metadata, it indicates that the source data corresponding to the target metadata has been cached in the cache service, and the target cache resource corresponding to the target metadata is cached in the cache service. In this case, the mounting situation is that the target mount path corresponding to the target cache resource exists in the cache routing table. When the metadata corresponding to any cache resource does not match the target metadata, it indicates that the source data corresponding to the target metadata has not been cached in the cache service, and the target cache resource corresponding to the target metadata is not cached in the cache service. In this case, the mounting situation is that the target mount path corresponding to the target cache resource does not exist in the cache routing table.

[0084] Step 103: Access the target cache resource or the source data corresponding to the target metadata based on an access policy matching the mounting situation.

[0085] In an embodiment of the present application, the client also periodically obtains routing rules from the server, and the routing rules store access policies corresponding to different mounting situations. The access policies are set to adapt to different types of availability zones and data clusters. In other words, the matching process of the access policy is only related to the mounting situation of the cache resource, and has nothing to do with the type of availability zone and data cluster. Therefore, different availability zones and data clusters can be applied to a set of routing rules, thereby achieving unified access to different availability zones and data clusters.

[0086] Different mounting situations may correspond to different access strategies, wherein, when the target cache resource corresponding to the target metadata is cached in the cache service, that is, when the mounting situation is that the target mount path corresponding to the target cache resource exists in the cache routing table, the target cache resource in the cache service may be directly accessed; when the target cache resource corresponding to the target metadata is not cached in the cache service, that is, when the mounting situation is that the target mount path corresponding to the target cache resource does not exist in the cache routing table, the source data corresponding to the target metadata in the underlying file system may be directly accessed. It is understandable that after directly accessing the source data corresponding to the target metadata in the underlying file system, the source data may be cached to obtain the target cache resource corresponding to the metadata in the cache service, and then when the client accesses the source data again, the target cache resource in the cache service may be directly accessed.

[0087] For example, the fields in the cache routing table may be as shown in Table 1.

[0088]

[0089]

[0090] Table 1

[0091] To summarize, the embodiments of the present application share the metadata corresponding to the cache resources between different availability zones, so that when a resource access request sent by a client is received, the mount path of the cache resource can be queried in the cache routing table storing the metadata based on the target metadata carried in the resource access request to understand the mount status of the cache resource, and then the applicable access strategy is selected for different mount statuses to access the cache resource or source data, so that in the process of accessing the cache resource, there is no need to pay attention to the differences between different availability zones, and only the metadata of the required access data needs to be specified. The data access process can be conveniently implemented through a set of unified access strategies, thereby improving the versatility of data access while simplifying the data access process, and improving the efficiency of data access to a certain extent.

[0092] Optionally, the step 103 includes:

[0093] Step 1031: when the mount condition is that the target mount path corresponding to the target cache resource exists in the cache routing table, determine that the access policy matching the mount condition is the first access policy, and convert the resource access request into an access request for the cache service;

[0094] Step 1032: Access the target cache resource based on a first access strategy; the first access strategy is to access the target cache resource in the available zone cache based on the queried target mount path.

[0095] Step 1033: when the mount condition is that the target mount path corresponding to the target cache resource does not exist in the cache routing table, determine that the access policy matching the mount condition is the second access policy, and convert the resource access request into an access request to the underlying file system;

[0096] Step 1034: In response to the access request to the underlying file system, based on the second access policy, access the source data corresponding to the target metadata; the second access policy is to access the source data corresponding to the target metadata in the underlying file system.

[0097] In an embodiment of the present application, according to different mounting situations, the user's resource access request can be converted into an access request for a cache service or an underlying file system. Accordingly, the client will access the cache service or the underlying file through the cache service client or the underlying file system (UFS, Under File System) client based on the converted access request to complete the access process for the cache resource or source data. If the target mount path corresponding to the target cache resource exists in the cache routing table, the resource access request can be converted into an access request for the cache service. Accordingly, the client will respond to the access request for the cache service and access the target cache resource in the cache service based on the first access policy. If the target mount path does not exist in the cache routing table, the resource access request can be converted into an access request for the underlying file system. Accordingly, the client will respond to the access request for the underlying file system and directly access the source data corresponding to the target metadata based on the second access policy.

[0098] Different mounting situations may correspond to different access policies. When the target mounting path corresponding to the target cache resource exists in the cache routing table, the access policy that matches the mounting situation is the first access policy; when the target mounting path corresponding to the target cache resource does not exist in the cache routing table, the access policy that matches the mounting situation is the second access policy. In other words, if the mounting situation is that the target mounting path corresponding to the target cache resource exists in the cache routing table, it means that the target cache resource corresponding to the target metadata can be accessed by accessing the cache service. Specifically, the target cache resource can be accessed based on the first access policy corresponding to the mounting situation. The first access policy can be to access the target cache resource in the available area cache based on the queried target mounting path. If the mounting situation is that the target mounting path does not exist in the cache routing table, it means that the source data corresponding to the target metadata is not cached in the cache service, that is, the cache service does not contain the target cache resource corresponding to the target metadata. At this time, the source data corresponding to the target metadata can be accessed by accessing the underlying file system. Specifically, the source data corresponding to the target metadata can be accessed based on the second access policy corresponding to the mounting situation. The second access policy can be to directly access the source data corresponding to the target metadata in the underlying file system.

[0099] Exemplarily, a comparison can be made between the storage path corresponding to the source data contained in the target metadata and the content of the field "path" corresponding to each cache resource in the cache routing table. When the storage path corresponding to the source data hits the content of the field "path", the mounting situation is characterized by the target mounting path corresponding to the target cache resource existing in the cache routing table; when the storage path corresponding to the source data does not hit the content of any field "path", the mounting situation is characterized by the target mounting path corresponding to the target cache resource not existing in the cache routing table.

[0100] The embodiment of the present application determines whether the target mount path corresponding to the target cache resource exists in a cache routing table shared by multiple different availability zones, and determines the storage location of the source data corresponding to the target cache resource or target metadata to be accessed, thereby achieving flexible selection of accessing the cache service or the underlying file system.

[0101] Optionally, when the access operation corresponding to the resource access request is a read operation, step 1031 includes:

[0102] Step 201: Obtain the first available zone where the source data corresponding to the target metadata in the underlying file system is located, the second available zone set where the target cache resource is located, and the third available zone where the client is located.

[0103] In an embodiment of the present application, when the access operation corresponding to the resource access request is a read operation, that is, the type of the resource access request is a resource read request, the third available zone (clientAZ) where the client is located is determined based on the client configuration information in the server. Exemplarily, the client configuration information may include available zone information corresponding to different IP network segments, and the client may obtain the client configuration information from the server, and determine the available zone corresponding to the client based on the client's IP as the third available zone.

[0104] If the source data in the underlying file system is cached, the cache routing table will contain the source data and the corresponding relationship between the cache resources and the available zones corresponding to the source data. Therefore, the client can obtain the cache routing table from the server and determine the available zone where the source data corresponding to the target metadata is located based on the cache routing table as the first available zone (dataAZ). In addition, the second available zone set where the target cache resource is located is determined based on the cache routing table. Since a source data can be cached in the available zone caches of different available zones, that is, for a source data, the cache service contains one or more cache resources corresponding to the source data, therefore, the second available zone set where the target cache resource is located can contain one or more second available zones (cacheAZ). Exemplarily, the target cache resource corresponding to the source data A is cached in the available zone cache of available zone A and the available zone cache of available zone B, then the second available zone set where the target cache resource is located contains available zone A and available zone B. Accordingly, the target cache resource corresponding to the source data A in the available zone cache of available zone A can be read through the cache service node corresponding to available zone A, or the target cache resource corresponding to the source data A in the available zone cache of available zone B can be read through the cache service node corresponding to available zone B.

[0105] Step 202: When the second available zone set includes the third available zone, determine a first cache service node from cache service nodes corresponding to the third available zone based on the target mount path and a consistent hashing algorithm.

[0106] In an embodiment of the present application, for a read operation of a cache resource, if the queried target cache resource is located in the available zone cache of the client, the target cache resource can be directly read based on the available zone cache of the cache service node in the available zone where the client is located. In the case where the third available zone is included in the second available zone set, the first cache service node can be determined from the cache service nodes corresponding to the third available zone by a consistent hashing algorithm based on the identifier (key) carried by the resource access request. Exemplarily, hash calculations can be performed on each cache service node corresponding to the third available zone based on a target hash function (for example, hash calculations are performed on the node identifiers corresponding to each cache service node based on the target hash function), and hash calculation results corresponding to each cache service node are obtained. The identifier carried by the resource access request is hashed based on the target hash function to obtain a target hash result. The cache service node corresponding to the hash calculation result corresponding to each cache service node and the target hash result is determined as the first cache service node. Among them, the identifier can be determined based on the mount path information of the target mount path, and the cache service node corresponding to the third available zone can be a cache service node contained in the available zone cache located in the third available zone. When each cache service node corresponding to each availability zone is started online, it will configure the availability zone information of its own cache service node, that is, the availability zone it is in will be used as its own cache service node availability zone (severAZ). For example, if cache service node worker-a is located in availability zone AZ-a, then its corresponding cache service node availability zone (severAZ) is AZ-a; if cache service node worker-b is located in availability zone AZ-b, then its corresponding cache service node availability zone (severAZ) is AZ-b. The client can obtain the cache service node availability zone corresponding to each cache service node through the cache service client, and then determine the cache service node corresponding to the third availability zone.

[0107] Exemplarily, a hash function can be used to calculate the hash value of the identifier carried in the resource access request, and based on the hash value, a mapped cache service node can be determined from the cache service nodes corresponding to the third available zone as the first cache service node. In this way, when the client subsequently reads the target cache resource based on the first cache service node, the target cache resource can be accessed nearby by selecting the cache service node in the available zone where the client is located, that is, the third available zone, which reduces the overhead to a certain extent.

[0108] Step 203: When the second availability zone set does not include the third availability zone but includes the first availability zone, determine the first cache service node from the cache service nodes corresponding to the first availability zone based on the target mount path and the consistent hashing algorithm.

[0109] In an embodiment of the present application, when the second availability zone set does not include the third availability zone but includes the first availability zone, the first cache service node can be determined from the cache service nodes corresponding to the first availability zone through a consistent hashing algorithm based on the identifier (key) carried in the resource access request. Among them, the identifier can be determined based on the mount path information of the target mount path, and the cache service node corresponding to the first availability zone can be a cache service node included in the availability zone cache located in the first availability zone. The client can obtain the cache service node availability zone corresponding to each cache service node through the cache service client, and then determine the cache service node corresponding to the first availability zone.

[0110] Step 204: Based on the first cache service node, a read operation is performed on the target cache resource in the cache in the available zone corresponding to the first cache service node.

[0111] In an embodiment of the present application, after determining the first cache service node, a read operation is performed on the target cache resource in the available zone cache corresponding to the first cache service node in the cache service based on the first cache service node. Exemplarily, when the first cache service node is determined from a cache service node in a third available zone, a read operation is performed on the target cache resource cached in the available zone cache in the third available zone based on the first cache service node; when the first cache service node is determined based on a cache service node in the first available zone, a read operation is performed on the target cache resource cached in the available zone cache in the first available zone based on the first cache service node.

[0112] The method further comprises:

[0113] Step 205: When the second available zone cache set does not include the third available zone and the first available zone, a read operation is performed on the source data corresponding to the target metadata in the underlying file system.

[0114] In the embodiment of the present application, when the third available zone and the first available zone are not included in the second available zone cache set, a read operation is performed directly on the source data corresponding to the target metadata in the underlying file system.

[0115] In an embodiment of the present application, when the access operation corresponding to the resource access request is a read operation, different access strategies are determined through the availability zone where the source data is located, the availability zone where the target cache resource is located, and the availability zone where the client is located, and the specific method of performing the read operation on the metadata corresponding to the target cache resource or the target metadata is judged, and then different routing strategies are executed for different mounting situations.

[0116] Exemplarily, the scenario routing rules for read operations may include:

[0117] Rule 1: When the third availability zone is included in the second availability zone set, the first cache service node is selected from the cache service nodes corresponding to the third availability zone based on consistent hashing, and the read operation on the target cache resource is completed based on the first cache service node.

[0118] Rule 2: When the first availability zone is included in the second availability zone set, the first cache service node is selected from the cache service nodes corresponding to the first availability zone based on consistent hashing, and the read operation on the target cache resource is completed based on the first cache service node.

[0119] Rule 3: In addition to the above rules 1 and 2, the source data corresponding to the target metadata is directly read in the underlying file system.

[0120] The above scenario routing rules are matched in sequence, and the matching is terminated when any scenario routing rule is matched.

[0121] Accordingly, examples are given based on different situations corresponding to the above scenario routing rules, as shown in Table 2:

[0122]

[0123]

[0124] Table 2

[0125] For example, assume that the user uses the client to read the file hdfs: / / az-b / data-b / read.txt, and the corresponding cache service storage address is alluxio: / / ebj@az1 / data-b / read.txt. The read operation process is as follows:

[0126] 1. When the "client" is created, it obtains the cache routing table and client configuration information from the server and caches them on the client. At the same time, it updates the data from the server every 5 minutes;

[0127] 2. The resource access request is processed by the routing client FileSystem proxy, calling the open("hdfs: / / az-b / data-b / read.txt") method;

[0128] 3. Call the open method to match the mount path from the cache service;

[0129] a. If there is a matching target mount path, determine whether to access the cache layer to obtain the file (based on the scenario routing rules of the read operation) based on the client information (the AZ to which the client belongs) and the cache routing table (cache resource group, cache AZ, client configuration);

[0130] 1) If a cache resource in a cache service is accessed, a read operation is performed on the target cache resource based on the selected first cache service node (rule 1 and rule 2);

[0131] 2) If the first cache service node is not selected, that is, the cache resource in the cache service is not accessed, the UFS FileSystem is used to directly read the UFS to obtain the file (Rule 3);

[0132] b. If there is no matching target mount path, it will be downgraded to UFS access, and the UFS FileSystem will be used to directly read the UFS to obtain the file.

[0133] Optionally, when the access operation corresponding to the resource access request is a write operation, step 1031 includes:

[0134] Step 301: When the write mode corresponding to the write operation is not the target write mode, obtain the first available area where the source data corresponding to the target metadata in the underlying file system is located; the target write mode is the underlying file system write mode.

[0135] In an embodiment of the present application, when the access operation corresponding to the resource access request is a write operation, that is, the type of the resource access request is a resource write request, the resource write request is parsed to determine the write mode (write_mode) corresponding to the write operation. When the write mode is not a target write mode, the first available area where the source data corresponding to the target metadata in the underlying file system is located is obtained. Among them, the write mode may include a write mode (THROUGH), a cache write mode (CACHE_THRGOUH), and an underlying file system write mode (UFS write mode), and the target write mode may be an underlying file system write mode (UFS write mode).

[0136] Step 302: Based on the target mount path and the consistent hashing algorithm, determine a second cache service node from cache service nodes corresponding to the first availability zone.

[0137] In an embodiment of the present application, the second cache service node can be determined from the cache service node corresponding to the first available zone through consistent hashing based on the identifier (key) carried by the resource write request. The identifier can be determined based on the mount path information of the target mount path, and the cache service node corresponding to the first available zone can be a cache service node included in the available zone cache located in the first available zone.

[0138] Step 303: Based on the second cache service node, in the cache of the available zone corresponding to the second cache service node, a write operation is performed on the target data carried in the resource access request according to the write mode corresponding to the write operation.

[0139] In an embodiment of the present application, after determining the second cache service node, based on the second cache service node, a write operation is performed on the target data carried by the resource access request according to the write mode corresponding to the write operation. Among them, the write mode corresponding to the write operation may include THROUGH and CACHE_THRGOUH. Exemplarily, in the case where the write mode is THROUGH, the target data is written to the underlying file system through the cache service, and after the write operation is performed, the cache routing table is updated so that the metadata of the target data is consistent with the metadata of the target data in the underlying file system. In the case where the write mode is CACHE_THRGOUH, the target data is written to the underlying file system through the cache service, and the target data is cached to the cache service, and at the same time, it is ensured that the metadata corresponding to the target data in each available zone remains consistent. Of course, the write mode at this time is only an exemplary description, and can be flexibly set according to actual needs, and is not limited here.

[0140] The method further comprises:

[0141] Step 304: When the write mode corresponding to the write operation is the target write mode, a write operation is performed on the target data carried in the resource access request in the underlying file system.

[0142] In an embodiment of the present application, when the write mode corresponding to the write operation is the target write mode, that is, when the write mode is UFS, the write operation is performed directly on the target data carried by the resource access request in the underlying file system.

[0143] The method further comprises:

[0144] Step 305: When the write mode corresponding to the write operation is not the target write mode and there is no cache service node in the cache of the first available zone, send access failure information for the resource access request.

[0145] In an embodiment of the present application, when the write mode corresponding to the write operation is not the target write mode, that is, the write mode is not the underlying file system write mode (UFS write mode) and there is no cache service node in the first available area cache, it indicates that the write operation cannot be performed through the cache service. Then, the write operation corresponding to this resource access request fails, and access failure information is sent for the resource access request.

[0146] Exemplarily, the scenario routing rules for write operations may include:

[0147] Rule 1: When the write mode corresponding to the write operation is the UFS write mode, the write operation is performed directly on the target data carried by the resource access request in the underlying file system.

[0148] Rule 2: When the write mode corresponding to the write operation is not the UFS write mode, a second cache service node is selected from the cache service nodes corresponding to the first availability zone based on consistent hashing, and a write operation is performed on the target data carried in the resource access request based on the second cache service node.

[0149] Rule 3: When the write mode corresponding to the write operation is not the UFS write mode and there is no cache service node in the first available zone, the target data carried by the resource access request is not allowed to be configured with cache, the write fails, and an access failure message is sent for the resource access request.

[0150] The above scenario routing rules are matched in sequence, and the matching is terminated when any scenario routing rule is matched.

[0151] Accordingly, examples are given based on different situations corresponding to the above scenario routing rules, as shown in Table 3:

[0152]

[0153]

[0154] Table 3

[0155] For example, assume that the user uses the "client" to write the file hdfs: / / az-b / data-b / write.txt, and the corresponding cache service storage address is alluxio: / / ebj@az1 / data-b / write.txt;

[0156] 1. The resource access request is processed by the client FileSystem proxy, calling the create("hdfs: / / az-b / data-b / write.txt") method;

[0157] 2. Call the create method to match the mount path from the cache service;

[0158] a. If there is a matching target mount path, determine the first available zone and execute the cache routing strategy based on the scenario routing rules of the write operation;

[0159] 1) If you access cache resources in the cache service, connect to the corresponding cache resource group through the "cache service client" to write files in CACHE or CACHE_THROUGH mode;

[0160] 2) If the cache resource in the cache service is not accessed, or the write mode is the UFS write mode, then go to step b;

[0161] b. If there is no matching target mount path or the write mode is UFS write mode, downgrade to UFS access and use UFS FileSystem to write directly to UFS.

[0162] In a possible implementation, if the mounted UFS is a non-data lake (iceberg) path, all write operations corresponding to resource write requests received by the client need to write data through the cache service; if the mounted UFS is a data lake (iceberg) path, all write operations corresponding to resource write requests received by the client need to write data through the cache service, or all read operations corresponding to resource read requests received by the client need to read data through the cache service. Because the data lake (iceberg) can be changed based on the delta mechanism, it will maintain the metadata of the latest version of the valid data file, and thus when SQL accesses the data lake (iceberg), it can ensure that the latest data file is obtained.

[0163] For example, the present application embodiment provides a specific schematic diagram of a data access method, such as Figure 3 As shown in the figure, there are three availability zones in the system architecture, namely AZ-a, AZ-b, and AZ-c; four clients, namely client-a, client-a2, client-b, and client-c; and three underlying file systems, namely HDFS-a, HDFS-b, and HDFS-c. The cache service is Alluxio-a, which is deployed across AZ-a and AZ-b, and ensures that the metadata of cache resources in the cache service are consistent. Assuming that the user uses client-a to read data data-b in HDFS-b, the steps of the data access method are as follows: First, the client obtains the cache routing table and client configuration information from the server-x at intervals (for example, 5 minutes), and determines that the availability zone clientAZ where the client client-a is located is AZ-a based on the client's own IP and client configuration information. Based on the cache routing table, it can be determined that the availability zone dataAZ where the data data-b is located is AZ-b, and the availability zones cacheAZ where the cache resources corresponding to the data data-b are located are AZ-a and AZ-b. In this way, when client-a reads data data-b in HDFS-b, it will read the cache resources corresponding to data data-b based on the cache service node worker-a in AZ-a.

[0164] Exemplarily, the data access method provided in the embodiment of the present application can be applied in the Trino query scenario for intelligent advertising bidding. Through the method provided in the embodiment of the present application, the query efficiency and query performance of users when querying advertising bidding information can be improved. In addition, the data access method provided in the embodiment of the present application can be applied in the BI (business intelligence) dashboard scenario. There is no need to synchronize data from other AZs to the AZ where the client is located. Business personnel can directly query data across AZs, meeting business needs while reducing network latency.

[0165] Figure 4 : is a structural diagram of a data access device provided in an embodiment of the present application, the device comprising:

[0166] Receiving module 401, used to receive a resource access request carrying target metadata;

[0167] The query module 402 is used to determine the mounting status of the target cache resource corresponding to the target metadata based on the cache routing table; the cache routing table includes: the corresponding relationship between the metadata corresponding to the cache resources in different available zones and the mounting path;

[0168] The access module 403 is used to access the target cache resource or the source data corresponding to the target metadata based on the access policy matching the mounting situation.

[0169] Optionally, the access module 403 is further configured to:

[0170] When the mount condition is that the target mount path corresponding to the target cache resource exists in the cache routing table, determining that the access policy matching the mount condition is the first access policy, and converting the resource access request into an access request to the cache service;

[0171] In response to the access request to the cache service, access the target cache resource based on the first access policy; the first access policy is to access the target cache resource in the available zone cache based on the queried target mount path;

[0172] When the mount situation is that the target mount path corresponding to the target cache resource does not exist in the cache routing table, determine that the access policy matching the mount situation is a second access policy, and convert the resource access request into an access request to the underlying file system;

[0173] In response to the access request to the underlying file system, the source data corresponding to the target metadata is accessed based on the second access policy; the second access policy is to access the source data corresponding to the target metadata in the underlying file system.

[0174] Optionally, when the access operation corresponding to the resource access request is a read operation, the access module 403 is further configured to:

[0175] Obtaining the first available zone where the source data corresponding to the target metadata in the underlying file system is located, the second available zone set where the target cache resource is located, and the third available zone where the client is located;

[0176] In a case where the second available zone set includes the third available zone, determining a first cache service node from cache service nodes corresponding to the third available zone based on the target mount path and a consistent hashing algorithm;

[0177] In a case where the second availability zone set does not include the third availability zone but includes the first availability zone, determining a first cache service node from cache service nodes corresponding to the first availability zone based on the target mount path and a consistent hashing algorithm;

[0178] Based on the first cache service node, performing a read operation on the target cache resource in the cache of the available zone corresponding to the first cache service node;

[0179] In a case where the second available zone cache set does not include the third available zone and the first available zone, a read operation is performed on the source data corresponding to the target metadata in the underlying file system.

[0180] Optionally, when the access operation corresponding to the resource access request is a write operation, the access module 403 is further configured to:

[0181] In the case where the write mode corresponding to the write operation is not the target write mode, obtaining a first available area where the source data corresponding to the target metadata in the underlying file system is located; the target write mode is the underlying file system write mode;

[0182] Based on the target mount path and the consistent hashing algorithm, determine a second cache service node from cache service nodes corresponding to the first availability zone;

[0183] Based on the second cache service node, in the cache of the available zone corresponding to the second cache service node, a write operation is performed on the target data carried by the resource access request according to the write mode corresponding to the write operation;

[0184] When the write mode corresponding to the write operation is the target write mode, a write operation is performed on the target data carried in the resource access request in the underlying file system.

[0185] Optionally, the access module 403 is further configured to:

[0186] When the write mode corresponding to the write operation is not the target write mode and there is no cache service node in the cache of the first available zone, access failure information is sent for the resource access request.

[0187] The embodiment of the present application caches cache resources in different availability zones, so that when a resource access request sent by a client is received, the mount path of the cache resource can be queried in the cache routing table storing the metadata based on the target metadata carried in the resource access request to know the mount status of the cache resource, and then the applicable access strategy is selected for different mount statuses to access the cache resource or source data, so that in the process of accessing the cache resource, there is no need to pay attention to the differences between different availability zones, and only the metadata of the required access data needs to be specified. The data access process can be conveniently implemented through a set of unified access strategies, thereby improving the versatility of data access while simplifying the data access process, and improving the efficiency of data access to a certain extent.

[0188] As for the above-mentioned server embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0189] The present application also provides an electronic device, such as Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0190] Memory 503, used for storing computer programs;

[0191] The processor 501 is used to implement the steps of any of the above data access methods when executing the program stored in the memory 503.

[0192] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0193] The communication interface is used for communication between the above terminal and other devices.

[0194] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Alternatively, the memory may also be at least one storage device located away from the aforementioned processor.

[0195] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0196] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is executed on a computer, the computer executes the data access method described in any one of the above embodiments.

[0197] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the data access method described in any one of the above embodiments.

[0198] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0199] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0200] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0201] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A data access method, characterized in that: include: receiving a resource access request carrying target metadata; Based on the cache routing table, determining the mounting status of the target cache resource corresponding to the target metadata; The cache routing table includes: a correspondence between metadata corresponding to cache resources in different availability zones and mount paths; The target cache resource or the source data corresponding to the target metadata is accessed based on an access policy matching the mounting situation.

2. The method according to claim 1, characterized in that The accessing the target cache resource or the source data corresponding to the target metadata based on the access policy matching the mounting situation includes: When the mount condition is that the target mount path corresponding to the target cache resource exists in the cache routing table, determining that the access policy matching the mount condition is the first access policy, and converting the resource access request into an access request to the cache service; In response to the access request to the cache service, access the target cache resource based on the first access policy; the first access policy is to access the target cache resource in the available zone cache based on the queried target mount path; When the mount condition is that the target mount path corresponding to the target cache resource does not exist in the cache routing table, determine that the access policy matching the mount condition is a second access policy, and convert the resource access request into an access request to the underlying file system; In response to the access request to the underlying file system, the source data corresponding to the target metadata is accessed based on the second access policy; the second access policy is to access the source data corresponding to the target metadata in the underlying file system.

3. The method according to claim 2, characterized in that When the access operation corresponding to the resource access request is a read operation, accessing the target cache resource in the available zone cache based on the queried target mount path includes: Obtaining the first available zone where the source data corresponding to the target metadata in the underlying file system is located, the second available zone set where the target cache resource is located, and the third available zone where the client is located; In a case where the second available zone set includes the third available zone, determining a first cache service node from cache service nodes corresponding to the third available zone based on the target mount path and a consistent hashing algorithm; In a case where the second availability zone set does not include the third availability zone but includes the first availability zone, determining a first cache service node from cache service nodes corresponding to the first availability zone based on the target mount path and a consistent hashing algorithm; Based on the first cache service node, performing a read operation on the target cache resource in the cache of the available zone corresponding to the first cache service node; The method further comprises: In a case where the second available zone cache set does not include the third available zone and the first available zone, a read operation is performed on the source data corresponding to the target metadata in the underlying file system.

4. The method according to claim 2, characterized in that: When the access operation corresponding to the resource access request is a write operation, accessing the target cache resource in the available zone cache based on the queried target mount path includes: In the case where the write mode corresponding to the write operation is not the target write mode, obtaining a first available area where the source data corresponding to the target metadata in the underlying file system is located; the target write mode is the underlying file system write mode; Based on the target mount path and the consistent hashing algorithm, determine a second cache service node from cache service nodes corresponding to the first availability zone; Based on the second cache service node, in the cache of the available zone corresponding to the second cache service node, a write operation is performed on the target data carried by the resource access request according to the write mode corresponding to the write operation; The method further comprises: When the write mode corresponding to the write operation is the target write mode, a write operation is performed on the target data carried in the resource access request in the underlying file system.

5. The method according to claim 4, characterized in that The method further comprises: When the write mode corresponding to the write operation is not the target write mode and there is no cache service node in the cache of the first available zone, access failure information is sent for the resource access request.

6. A data access device, characterized in that: The device comprises: A receiving module, used for receiving a resource access request carrying target metadata; A query module, used to determine the mounting status of the target cache resource corresponding to the target metadata based on the cache routing table; the cache routing table includes: the corresponding relationship between the metadata corresponding to the cache resources in different available zones and the mounting path; An access module is used to access the target cache resource or the source data corresponding to the target metadata based on an access policy matching the mounting situation.

7. The device according to claim 6, characterized in that The access module is further used for: When the mount condition is that the target mount path corresponding to the target cache resource exists in the cache routing table, determining that the access policy matching the mount condition is the first access policy, and converting the resource access request into an access request to the cache service; In response to the access request to the cache service, access the target cache resource based on the first access policy; The first access strategy is to access the target cache resource in the available zone cache based on the queried target mount path; When the mount condition is that the target mount path corresponding to the target cache resource does not exist in the cache routing table, determine that the access policy matching the mount condition is a second access policy, and convert the resource access request into an access request to the underlying file system; In response to the access request to the underlying file system, accessing the source data corresponding to the target metadata based on the second access policy; The second access strategy is to access the source data corresponding to the target metadata in the underlying file system.

8. The device according to claim 7, characterized in that When the access operation corresponding to the resource access request is a read operation, the access module is further configured to: Obtaining the first available zone where the source data corresponding to the target metadata in the underlying file system is located, the second available zone set where the target cache resource is located, and the third available zone where the client is located; In a case where the second available zone set includes the third available zone, determining a first cache service node from cache service nodes corresponding to the third available zone based on the target mount path and a consistent hashing algorithm; In a case where the second availability zone set does not include the third availability zone but includes the first availability zone, determining a first cache service node from cache service nodes corresponding to the first availability zone based on the target mount path and a consistent hashing algorithm; Based on the first cache service node, performing a read operation on the target cache resource in the cache of the available zone corresponding to the first cache service node; In a case where the second available zone cache set does not include the third available zone and the first available zone, a read operation is performed on the source data corresponding to the target metadata in the underlying file system.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the data access method described in any one of claims 1 to 5 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data access method according to any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Cloud storage system, data read-write method, equipment and storage medium

    CN117240873A

  • Data access method and device, storage medium and electronic equipment

    CN117667879A

  • Smart Direct Access

    US20220092194A1