A data reading method and device for JuiceFS, electronic equipment and storage medium

Abstract

The application provides a data reading method and device for JuiceFS, electronic equipment and storage medium, and relates to the technical field of data management. In the application, in response to a data reading request initiated by a target client for a target task, the data reading request is analyzed to obtain a data access context of the target task; a target cache index matched with the data access context is selected from a preset cache index set; target data required for executing the target task is matched based on the target cache index to obtain a cache matching result; the cache matching result is used to indicate at least one storage location of the target data; a target storage location with the minimum time delay required for reading the target data is filtered from the at least one storage location indicated by the cache matching result, and the target data read from the target storage location is sent to the target client. In this way, flexible data reading near the target client is realized, and the cost of data reading is reduced.

Description

Technical Field

[0001] This application relates to the field of data management technology, and in particular to a data reading method, apparatus, electronic device and storage medium for JuiceFS. Background Technology

[0002] JuiceFS is a cloud-native, high-performance distributed file system consisting of three core components: a client, a metadata engine, and a data storage component. It employs an architecture that separates file data from metadata. The client is the core execution component of JuiceFS, initiating and executing all background tasks such as file reading and writing, fragment merging, and recycle bin expiration cleanup. It also supports multiple access methods. The metadata engine is specifically responsible for storing all of JuiceFS's metadata. The data storage component is responsible for storing file data, using object storage as the underlying storage to achieve persistent file data storage.

[0003] Currently, JuiceFS clients support local caching, which can reduce latency caused by repeatedly reading data from object storage. For example, local caching can improve read performance in scenarios with a single compute node or a small number of clients. However, existing caching strategies rely on fixed parameters (such as cache size, expiration time, or manual preheating lists), making it difficult to achieve flexible data retrieval from nearby locations and thus reduce data retrieval costs. Summary of the Invention

[0004] This application provides a data reading method, apparatus, electronic device, and storage medium for JuiceFS, which enables flexible data reading from the nearest location, thereby reducing the cost of data reading.

[0005] In a first aspect, embodiments of this application provide a data reading method for JuiceFS, the method comprising: In response to a data read request initiated by the target client for the target task, the data read request is parsed to obtain the data access context of the target task; From a pre-defined set of cache indexes, a target cache index that matches the data access context is selected; where each cache index in the cache index set is used to indicate the cache status of the data required to perform the corresponding task. Based on the target cache index, the target data required to execute the target task is matched against the data cache to obtain the cache matching result; the cache matching result is used to indicate at least one storage location of the target data. From at least one storage location indicated by the cache matching results, select the target storage location with the minimum latency required to read the target data, and send the target data read from the target storage location to the target client.

[0006] In an optional embodiment, prior to responding to a data read request initiated by the target client for the target task, the method further includes: The system retrieves the source read cost, task execution priority, and data popularity for each set of data stored in multiple object stores. Each source read cost represents the cost required to read the corresponding data from the corresponding object store, and each data popularity represents the frequency of data reading within a set time range. Based on the source read cost, task execution priority, and data popularity of each data point, the data storage location corresponding to each data point is determined. Set up separate cache indexes for multiple data storage locations, and save the multiple cache indexes to a cache index set.

[0007] In one optional embodiment, selecting a target cache index that matches the data access context from a preset set of cache indexes includes: Based on the data access context, determine the node cluster corresponding to the target client and the throughput, latency, and cost required to read the target data; Select at least one cache index corresponding to the node cluster from the cache index set, and obtain the target cache performance score based on throughput, latency and cost; The target cache index is determined based on at least one cache index that includes a cache performance score greater than the target cache performance score.

[0008] In one optional embodiment, data cache matching is performed on the target data required to perform the target task based on the target cache index to obtain the cache matching result, including: Based on the data version number included in the target cache index, the target task is matched against the local cache to obtain the first matching sub-result; the first matching sub-result indicates whether the target local cache that caches the target data was successfully matched. If the first matching sub-result indicates that the target local cache was not successfully matched, then the target task is subjected to distributed cache matching based on the node status of each cache node included in the target cache index to obtain the second matching sub-result; the second matching sub-result indicates whether the target cache node for caching the target data was successfully matched. The cached matching result is obtained based on the first and second matching sub-results.

[0009] In an optional embodiment, after performing local cache matching on the target task based on the data version number included in the target cache index to obtain a first matching sub-result, the method further includes: If the first matching sub-result indicates a successful match with the target local cache, then the first matching sub-result will be used as the cache matching result.

[0010] In one optional embodiment, selecting the target storage location with the lowest latency required to read the target data from at least one storage location indicated by the cache matching results includes: If the cache matching result indicates that the target local cache of the target data has been successfully matched, then the target storage location is determined to be the target local cache; If the cache matching result indicates that the target local cache did not match successfully, but the target cache node successfully matched the target data, then the target storage location is determined to be the target cache node; If the cache matching result indicates that the target local cache and the target cache node were not successfully matched, then the target storage location is determined to be the target object storage included in the target cache index.

[0011] In an optional embodiment, the method further includes: If the number of clients reading the target data at the current moment reaches the set threshold, and the target data is cached in the target cache node, then while instructing some of the multiple clients reading the target data to read the target data from the target object storage, the target data will be cached in the idle cache node. After the target data is cached in the idle cache node, another group of clients among multiple clients is instructed to read the target data from both the target cache node and the idle cache node.

[0012] Secondly, embodiments of this application also provide a data reading device for JuiceFS, the device comprising: The request parsing module is used to respond to data read requests initiated by the target client for the target task, parse the data read requests, and obtain the data access context of the target task; The index selection module is used to select a target cache index that matches the data access context from a preset cache index set; wherein each cache index in the cache index set is used to indicate the cache status of the data required to perform the corresponding task; The cache matching module is used to perform data cache matching on the target data required to execute the target task based on the target cache index, and obtain the cache matching result; the cache matching result is used to indicate at least one storage location of the target data; The data reading module is used to filter out the target storage location with the minimum latency required to read the target data from at least one storage location indicated by the cache matching result, and send the target data read from the target storage location to the target client.

[0013] In an optional embodiment, before responding to a data read request initiated by the target client for the target task, the index selection module is further configured to: The system retrieves the source read cost, task execution priority, and data popularity for each set of data stored in multiple object stores. Each source read cost represents the cost required to read the corresponding data from the corresponding object store, and each data popularity represents the frequency of data reading within a set time range. Based on the source read cost, task execution priority, and data popularity of each data point, the data storage location corresponding to each data point is determined. Set up separate cache indexes for multiple data storage locations, and save the multiple cache indexes to a cache index set.

[0014] In an optional embodiment, when selecting a target cache index that matches the data access context from a preset set of cache indexes, the index selection module is specifically used for: Based on the data access context, determine the node cluster corresponding to the target client and the throughput, latency, and cost required to read the target data; Select at least one cache index corresponding to the node cluster from the cache index set, and obtain the target cache performance score based on throughput, latency and cost; The target cache index is determined based on at least one cache index that includes a cache performance score greater than the target cache performance score.

[0015] In an optional embodiment, when performing data cache matching on the target data required to execute the target task based on the target cache index to obtain the cache matching result, the cache matching module is specifically used for: Based on the data version number included in the target cache index, the target task is matched against the local cache to obtain the first matching sub-result; the first matching sub-result indicates whether the target local cache that caches the target data was successfully matched. If the first matching sub-result indicates that the target local cache was not successfully matched, then the target task is subjected to distributed cache matching based on the node status of each cache node included in the target cache index to obtain the second matching sub-result; the second matching sub-result indicates whether the target cache node for caching the target data was successfully matched. The cached matching result is obtained based on the first and second matching sub-results.

[0016] In an optional embodiment, after performing local cache matching on the target task based on the data version number included in the target cache index to obtain a first matching sub-result, the cache matching module is further configured to: If the first matching sub-result indicates a successful match with the target local cache, then the first matching sub-result will be used as the cache matching result.

[0017] In an optional embodiment, when selecting the target storage location with the lowest latency required to read the target data from at least one storage location indicated by the cache matching results, the data reading module is specifically configured to: If the cache matching result indicates that the target local cache of the target data has been successfully matched, then the target storage location is determined to be the target local cache; If the cache matching result indicates that the target local cache did not match successfully, but the target cache node successfully matched the target data, then the target storage location is determined to be the target cache node; If the cache matching result indicates that the target local cache and the target cache node were not successfully matched, then the target storage location is determined to be the target object storage included in the target cache index.

[0018] In an optional embodiment, the data reading module is further configured to: If the number of clients reading the target data at the current moment reaches the set threshold, and the target data is cached in the target cache node, then while instructing some of the multiple clients reading the target data to read the target data from the target object storage, the target data will be cached in the idle cache node. After the target data is cached in the idle cache node, another group of clients among multiple clients is instructed to read the target data from both the target cache node and the idle cache node.

[0019] Thirdly, embodiments of this application provide an electronic device, including: processor; Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the data reading method for JuiceFS as described in the first aspect.

[0020] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the data reading method for JuiceFS as described in the first aspect.

[0021] Fifthly, this application provides a computer program product that, when invoked by a computer, causes the computer to execute the data reading method steps for JuiceFS as described in the first aspect.

[0022] The beneficial effects of this application are as follows: In the data reading method for JuiceFS provided in this application embodiment, the data reading request initiated by the target client for the target task is parsed to obtain the data access context of the target task. Then, a target cache index matching the data access context is selected from a preset cache index set. Each cache index in the cache index set indicates the cached state of the data required to execute the corresponding task. Further, data cache matching is performed on the target data required to execute the target task based on the target cache index to obtain a cache matching result. The cache matching result indicates at least one storage location of the target data. Finally, the target storage location with the lowest latency required to read the target data is selected from the at least one storage location indicated by the cache matching result, and the target data read from the target storage location is sent to the target client. In this way, by pre-setting cache indexes for different data storage locations, target data can be read from the storage location with the lowest latency based on the target cache index associated with the data access context of the target task, achieving flexible data reading from the nearest location and thus reducing the cost of data reading.

[0023] Furthermore, other features and advantages of this application will be set forth in the following description and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described herein are used to provide a further understanding of this application, constitute a part of this application, and do not constitute an improper limitation of this application. In the accompanying drawings: Figure 1 A schematic diagram illustrating the implementation process of a data reading method for JuiceFS provided in this application embodiment; Figure 2 This is a logical diagram illustrating the selection of a cache index, provided in an embodiment of this application.

[0025] Figure 3 This is a schematic diagram illustrating the implementation process of a data cache matching method provided in an embodiment of this application.

[0026] Figure 4 This is a schematic diagram of a system architecture for data reading via JuiceFS, provided as an embodiment of this application.

[0027] Figure 5 This is a schematic diagram of a data reading device for JuiceFS provided in an embodiment of this application.

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0030] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0031] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0032] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0033] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0034] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0035] (1) Local cache: refers to the data block cache stored on the local disk or in memory of the client node in JuiceFS, which is used to reduce the latency of repeated reads.

[0036] (2) Distributed caching: refers to the shared caching capability composed of multiple cache nodes, which enables multiple clients to reuse cached data blocks.

[0037] (3) Cache Group: refers to a collection of cache nodes organized by business, region, cluster or task type to support local reading and cache reuse.

[0038] (4) Read from source: refers to reading data from object storage or underlying data medium when the cache is missed or the cache is unavailable.

[0039] Based on the above explanations of terms and related terminology, the design concept of the embodiments of this application will be briefly introduced below: After a unified view of a hybrid cloud data lake is established, users can access data through a unified path, but the data may actually reside in remote object storage, cross-cloud buckets, or storage media in different regions. Big data queries, artificial intelligence (AI) training, and batch analysis tasks often repeatedly read the same data. If cross-cloud origin access is required every time, it will result in high latency, bandwidth costs, and pressure on object storage requests.

[0040] Currently, JuiceFS clients support local caching, which can reduce latency caused by repeatedly reading data from object storage. For example, local caching can improve read performance in scenarios with a single compute node or a small number of clients.

[0041] However, existing caching strategies rely on fixed parameters (such as cache size, expiration time, or manual preheating list), making it difficult to achieve flexible data retrieval based on proximity, thereby reducing the cost of data retrieval.

[0042] To address or improve the aforementioned problems, this application provides a data reading method for JuiceFS, which specifically includes: First, in response to a data reading request initiated by a target client for a target task, parsing the data reading request to obtain the data access context of the target task; next, selecting a target cache index that matches the data access context from a preset cache index set; wherein each cache index in the cache index set is used to indicate the cached state of the data required to execute the corresponding task; further, performing data cache matching on the target data required to execute the target task based on the target cache index to obtain a cache matching result; the cache matching result is used to indicate at least one storage location of the target data; finally, selecting the target storage location with the lowest latency required to read the target data from the at least one storage location indicated by the cache matching result, and sending the target data read from the target storage location to the target client. In this way, by using cache indexes pre-set for different data storage locations, target data can be read from the storage location with the lowest latency based on the target cache index associated with the data access context of the target task, achieving flexible data reading from the nearest location and thus reducing the cost of data reading.

[0043] In particular, the preferred embodiments of this application will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments of this application and the features in the embodiments can be combined with each other unless otherwise specified.

[0044] See Figure 1 The diagram illustrates the implementation flow of a data reading method for JuiceFS provided in this application embodiment. The execution entity is a server, which can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This application embodiment does not limit this specific implementation. The detailed implementation flow of the method is as follows: S101: In response to a data read request initiated by the target client for the target task, parse the data read request to obtain the data access context of the target task.

[0045] For example, during step S101, the data read request initiated by the target client for the target task can be parsed using the access context model in the server to obtain the data access context of the target task. Optionally, the aforementioned data access context may include fields such as: client identifier (clientId), cluster identifier (clusterId), region / availability zone (region), job type (jobType), user identifier (userId), access protocol (protocol), target path (targetPath), read range (readRange), expected throughput (expectedThroughput), latency sensitivity (latencySensitivity), and cost sensitivity (costSensitivity).

[0046] The client identifier uniquely identifies the client initiating the data read request, facilitating the tracking and analysis of the request source. The cluster identifier identifies the cluster or service instance where the data read request resides, enabling load balancing, failover, and monitoring in the distributed system. The region / availability zone refers to the geographical area where the data read request is processed, helping to optimize latency. The task type indicates the type or nature of the data read request, such as read operation, write operation, batch processing, etc., allowing subsequent system selection of appropriate processing logic and resources. The user identifier identifies the user initiating the data query request, which is crucial for access control and audit logs, ensuring that only authorized users can perform specific operations. The access protocol specifies the communication protocol used, such as Hypertext Transfer Protocol (HTTP) or Google Remote Procedure Call (GRP). (e.g., call, gRPC) helps in selecting the correct protocol processing method and verifying the correctness of the protocol; the target path specifies the target path or resource for the data query request so that the data query request can be completed through the correct processing component or service; the read range refers to specifying the range or conditions of data to be read in the read operation to optimize data access and improve performance; the expected throughput characterizes the expected throughput or processing speed, which helps the system optimize resource allocation and ensure that performance can be maintained under high load; latency sensitivity refers to the sensitivity of the data query request to latency, and high sensitivity may mean that a faster processing path or resource needs to be selected; cost sensitivity refers to the sensitivity of the data query request to cost (e.g., fees, energy consumption, etc.), which helps in making a balance decision between cost and performance.

[0047] S102: Select the target cache index that matches the data access context from the preset cache index set.

[0048] Each cache index in the aforementioned cache index set is used to indicate the cache status of the data required to perform the corresponding task, such as whether the data is cached or the data cache type (e.g., local cache or distributed cache).

[0049] For example, during step S102, the cache index matching of the data access context can be performed using the cache index model in the server, and the target cache index can be selected from a preset cache index set. Optionally, each cache index may include fields such as: namespace identifier (namespaceId), data path (dataPath), chunk identifier (chunkId), chunk version number (chunkVersion), local cache state (localCacheState), cache group identifier (cacheGroupId), cache node list (cacheNodeList), source storage location (sourceStorage), checksum (checksum), last access time (lastAccessTime), hot score (hotScore), and expiration time (expireTime).

[0050] Namespace identifiers are used to logically isolate data domains from different tenants or projects. Each namespace can be considered an independent cache area used to isolate data from different applications or services. For example, namespace identifiers might be "user_data" or "product_catalog". Data path indicates the path or key name of cached data, typically used to uniquely identify cached objects. Shard identifiers are used when processing large objects or data shards; each shard has a unique identifier. For example, a shard identifier could be "chunk_001", representing a shard of a large object. Shard version number records the data version number, handling data updates and consistency. For example, a shard version number of "v1.0". Local cache status indicates the status of the local cache, such as whether it has been hit or expired. Cache group identifier identifies a group of multiple cache nodes to ensure subsequent load balancing and failover. Cache node list indicates all nodes participating in caching operations. For example, a cache node list might be: ["node1", "node2", "node3"]. Source storage location refers to the original location or source of the data, i.e., the object storage of the data. Checksums are used to verify data integrity, determining that the data has not been tampered with during transmission or storage. Last access time refers to the time when a cached item or data was last accessed. Popularity score measures the frequency of a cached item, used to determine whether data should be prioritized for loading or eviction; it can also be called data popularity or historical access popularity, which is not limited in this embodiment. For example, a popularity score of 90 indicates that the data is frequently accessed or queried. Expiration time refers to the period after which a cached item or data is considered invalid or cleared.

[0051] It should be noted that the shard version number and / or checksum can be verified before reading the cache. If the cache is found to be expired or corrupted, the cache is skipped and the source is retrieved, and the cache copy is marked as invalid.

[0052] To improve data retrieval efficiency, different cache indexes can be pre-set for different tasks or data. In one optional implementation, before responding to a data retrieval request initiated by a target client for a target task, the server can first obtain the source-end read cost, task execution priority, and data popularity for each piece of data stored in multiple object stores. Here, each source-end read cost represents the cost required to read the corresponding data from the corresponding object store, and each data popularity represents the read frequency of the corresponding data within a set time range; the higher the data popularity, the higher the read frequency.

[0053] Next, the server can determine the data storage location (e.g., local cache, cache node, or object storage) for each of the multiple data points based on the source read cost, task execution priority, and data popularity.

[0054] Optionally, the server can normalize the source read cost, task execution priority, and data popularity corresponding to multiple data points to obtain normalized source read cost, normalized task execution priority, and normalized data popularity. Combined with the priority cache weights corresponding to the source read cost, task execution priority, and data popularity, a priority cache score is obtained for each data point. The data points are then sorted by priority cache score from high to low to obtain the priority cache order. Finally, the data storage location is determined based on the priority cache order.

[0055] Taking 20 data items as an example, after obtaining the priority cache order of the 20 data items, the server can load the first 5 data items (i.e., the 5 data items with the highest priority cache scores) from their respective object storage into the local cache, and load the 6th to 12th data items (i.e., the 6th to 12th data items with relatively high priority cache scores) from their respective object storage into the cache nodes. The last 8 data items (i.e., the 8 data items with the lowest priority cache scores) do not need to be loaded into the local cache or cache nodes. For example, for cold data or data scanned once, only streaming reading is required, without loading into the local cache or cache nodes.

[0056] In addition, the server can also combine data popularity, file size, task type and cache capacity to decide whether to populate the local cache or the distributed cache (e.g., node cache).

[0057] Ultimately, the server sets up separate cache indexes for multiple data storage locations and saves these cache indexes into a cache index set. This pre-setting of different cache indexes for different data not only improves data retrieval efficiency but also determines data storage locations based on source read costs, task execution priorities, and data popularity, avoiding the problem of low-value data being pre-warmed and crowding out high-value cache (e.g., prioritizing loading lower-rated data into local or node caches).

[0058] Based on the above approach, for large-scale query, training, or periodic tasks that are about to be executed, the server can preload data with higher scores into the local cache or cache nodes included in the cache group according to the priority cache score.

[0059] It should be noted that the server can generate a cache policy version (cachePolicyVersion) based on path weight, preheating rules, eviction rules, cache group (or cache node) selection rules, and fallback rules. For tasks of the same type (e.g., the same training or computation job), the same policy version can be consistently used to avoid performance fluctuations caused by policy changes during task execution, which could result in some clients reading local caches while others frequently read object storage.

[0060] The server can generate cache preheating tasks based on the tasks to be run by the scheduling platform, historical access sequences, table partition popularity, and user-submitted training datasets. The preheating task includes fields such as target path, cache range (chunkRange), target cache group (targetCacheGroup), priority, expiration time, and bandwidth limit (bandwidthLimit). The preheating process is limited by bandwidth and capacity, but does not affect high-priority online reads.

[0061] The cache eviction mechanism (or eviction rules) is based not only on the most recent access time (i.e., the last access time mentioned above), but also on factors such as popularity score, tenant priority, task type, cache size, source read cost, and refill cost. This cache eviction mechanism increases the retention priority of data blocks with high cross-cloud origin access costs and frequent access by recent training tasks; while decreasing the retention priority of data blocks that are scanned only once, have low popularity, or can be retrieved from the origin at low cost.

[0062] The fallback rule is used to automatically switch to the next candidate cache or source object storage when local cache read fails, cache group is unreachable, cache verification fails, or cache node is overloaded, so as to ensure that data reading is successful.

[0063] Further, see Figure 2 As shown, during step S102, the server can determine the node cluster corresponding to the target client and the throughput, latency, and cost required to read the target data based on the data access context. It should be understood that the latency required to read the target data is determined based on the latency sensitivity included in the data access context, and the cost required to read the target data is determined based on the cost sensitivity included in the data access context.

[0064] Next, the server can select at least one cache index corresponding to the node cluster from the cache index set, and obtain a target cache performance score based on throughput, latency, and cost. Optionally, the target cache performance score can be obtained by weighted summation of throughput, latency, and cost.

[0065] Ultimately, after obtaining the target cache performance score, the server can determine the target cache index based on at least one cache index that includes a cache index with a cache performance score greater than the target cache performance score.

[0066] It should be noted that if there are multiple cache indexes with a cache performance score greater than the target cache performance score, a cache index can be randomly selected as the target cache index. Alternatively, the server can select the cache performance score closest to the target cache performance score from among the multiple cache performance scores greater than the target cache performance score, and then use the cache index corresponding to that cache performance score as the target cache index. This embodiment of the application does not limit this approach.

[0067] S103: Based on the target cache index, perform data cache matching on the target data required to execute the target task, and obtain the cache matching result.

[0068] The cache matching results described above can be used to indicate at least one storage location for the target data. This at least one storage location includes the object storage corresponding to the target data. In other words, there are three specific scenarios for at least one storage location: local cache and object storage, node cache and object storage, and object storage itself.

[0069] In one alternative implementation, during step S103, the server executes the reference. Figure 3 The method steps shown are as follows: S301: Perform local cache matching on the target task based on the data version number included in the target cache index to obtain the first matching sub-result.

[0070] The first matching sub-result described above indicates whether the target local cache of the target data was successfully matched.

[0071] S302: If the first matching sub-result indicates that the target local cache was not successfully matched, then the target task is matched in a distributed cache based on the node status of each cache node included in the target cache index to obtain the second matching sub-result.

[0072] The second matching sub-result described above indicates whether the target cache node for the cached target data was successfully matched.

[0073] S303: Obtain the cached matching result based on the first and second matching sub-results.

[0074] At this point, the cache matching result indicates that the target local cache that failed to match (or missed) the target data, but the target cache node that successfully matched (or hit) the target data, indicating the target cache node storing the target data and the target object storage. Alternatively, the cache matching result indicates that the target local cache and the target cache node failed to match (or missed) the target data, indicating the target object storage storing the target data.

[0075] Optionally, after the server performs local cache matching on the target task based on the data version number included in the target cache index and obtains the first matching sub-result, if the first matching sub-result indicates a successful match with the target local cache, the first matching sub-result can be directly used as the cache matching result.

[0076] At this point, the cache matching result indicates a successful match (or hit) with the target local cache, indicating the target local cache, target cache node, and target object storage where the target data is stored. Of course, caching of the target data can employ only one method to conserve cache resources; therefore, the cache matching result may only indicate the target cache node and target object storage.

[0077] S104: From at least one storage location indicated by the cache matching result, select the target storage location with the minimum latency required to read the target data, and send the target data read from the target storage location to the target client.

[0078] Understandably, the closer the storage location is to the target client, the faster the data can be read. Therefore, the latency of reading target data from the three storage locations, from smallest to largest, is: target local cache, target cache node, and target object storage. Thus, when executing step S104, if the cache matching result indicates a successful match with the target local cache containing the target data, the server can determine the target storage location as the target local cache; if the cache matching result indicates a failure to match the target local cache but a successful match with the target cache node containing the target data, the server can determine the target storage location as the target cache node; if the cache matching result indicates a failure to match either the target local cache or the target cache node, the server can determine the target storage location as the target object storage included in the target cache index.

[0079] Using the above-described data read path selection rules, priority is given to local caches that have passed verification. If the local cache is not found, a cache group (or multiple cache nodes) in the same data center or region (e.g., the same node cluster) is selected. If multiple cache nodes are found, they are sorted by latency, load, and health status. If no cache node is found, the data is retrieved from object storage. Optionally, the data read path selection results and policy version are written to the audit log for subsequent analysis.

[0080] In an alternative implementation, if the number of clients reading the target data at the current moment reaches a set threshold, and the target data is cached in the target cache node, the server can simultaneously instruct some of the multiple clients reading the target data to read the target data from the target object storage while caching the target data in the idle cache node. After the target data is cached in the idle cache node, the server can instruct another portion of the multiple clients to read the target data from both the target cache node and the idle cache node.

[0081] Using the concurrent read control described above, when multiple clients simultaneously request the same uncached data, the server can select one or a small number of clients to perform origin fetch (i.e., read data from object storage), while the remaining clients wait or read from the shared cache that is being filled (e.g., cache nodes), thus avoiding cache breakdown. Optionally, the server can also record information such as chunkId, leaderClient, waitingClients, fillState, and timeout during the concurrent read control process, allowing clients to independently fetch from the origin after a timeout, ensuring that reads are not blocked by a single point of contention.

[0082] Furthermore, after the reading is completed, the server can record the actual reading path, hit level, reading time, number of bytes retrieved from the source, number of bytes filled in the cache, number of failures, and number of object storage requests, etc., and adjust the warm-up list, weight parameters, eviction priority, and hotspot threshold, etc., based on the feedback. This application embodiment does not specifically limit this.

[0083] Based on the data reading method for JuiceFS described in steps S101 to S104 above, refer to... Figure 4 The diagram illustrates a system architecture for data reading via JuiceFS, as provided in this application embodiment. This system architecture comprises a read request layer, a cache decision layer, and a data source layer. The read request layer includes a task execution module (e.g., a computation task), a JuiceFS client (e.g., a target client), and an access context identification module. The cache decision layer includes a local cache hit determination module, a distributed cache query module, a read path selection module, and an effect feedback module. The data source layer includes a local cache (e.g., a disk cache), a distributed cache (e.g., cache nodes), and object storage.

[0084] The access context identification module is used to obtain information such as the target client's location, computing cluster, task type, user, target path, file size, and historical popularity. The read path selection module is used to determine whether a local cache matching the target data exists based on a preset mapping relationship between data and the local cache. The distributed cache query module is used to determine whether a cache node matching the target data exists based on a preset mapping relationship between data and cache nodes. The read path selection module is used to select local cache read, cache node read, or origin read (e.g., object storage read) based on local cache hit results, distributed cache hit results, cache node status, network distance (e.g., latency), data popularity, and object storage status. For example, for hot data, it can be preferentially read or preheated to the local cache or the nearest cache group (or cache node). It can be understood that when the cache misses or the cache is unavailable, data is read from object storage and written to the local cache or cache node according to a strategy. The performance feedback module records cache hit rate, read time, number of origin pulls, failure rate, cache fill volume, and object storage request volume, and feeds these metrics back to cache preheating, eviction, cache group (or cache node) selection, and read path weight adjustment.

[0085] Optionally, the cache decision layer may also include an exception handling module, etc., which is not limited in this embodiment. The exception handling module is responsible for rollback when cache read fails, verification fails, cache node is unreachable, and source object storage fails.

[0086] In summary, the data reading method for JuiceFS provided in this application embodiment parses the data reading request initiated by the target client for the target task to obtain the data access context of the target task. Then, a target cache index matching the data access context is selected from a preset cache index set. Each cache index in the cache index set indicates the cached state of the data required to execute the corresponding task. Further, data cache matching is performed on the target data required to execute the target task based on the target cache index to obtain a cache matching result. The cache matching result indicates at least one storage location of the target data. Finally, the target storage location with the lowest latency required to read the target data is selected from the at least one storage location indicated by the cache matching result, and the target data read from the target storage location is sent to the target client. Thus, by using cache indexes pre-set for different data storage locations, target data can be read from the storage location with the lowest latency based on the target cache index associated with the data access context of the target task, achieving flexible data retrieval based on proximity and thereby reducing the cost of data reading.

[0087] Furthermore, based on the same technical concept, embodiments of this application also provide a data reading device for JuiceFS, which can be used to implement the above-described method flow of embodiments of this application. For example, see [link to relevant documentation]. Figure 5 As shown, the data reading device 500 for JuiceFS may include: a request parsing module 501, an index selection module 502, a cache matching module 503, and a data reading module 504, wherein: The request parsing module 501 is used to respond to the data read request initiated by the target client for the target task, parse the data read request, and obtain the data access context of the target task; The index selection module 502 is used to select a target cache index that matches the data access context from a preset cache index set; wherein each cache index in the cache index set is used to indicate the cache status of the data required to perform the corresponding task; The cache matching module 503 is used to perform data cache matching on the target data required to execute the target task based on the target cache index, and obtain the cache matching result; the cache matching result is used to indicate at least one storage location of the target data. The data reading module 504 is used to filter out the target storage location with the minimum latency required to read the target data from at least one storage location indicated by the cache matching result, and send the target data read from the target storage location to the target client.

[0088] In an optional embodiment, before responding to a data read request initiated by the target client for the target task, the index selection module 502 is further configured to: The system retrieves the source read cost, task execution priority, and data popularity for each set of data stored in multiple object stores. Each source read cost represents the cost required to read the corresponding data from the corresponding object store, and each data popularity represents the frequency of data reading within a set time range. Based on the source read cost, task execution priority, and data popularity of each data point, the data storage location corresponding to each data point is determined. Set up separate cache indexes for multiple data storage locations, and save the multiple cache indexes to a cache index set.

[0089] In an optional embodiment, when selecting a target cache index that matches the data access context from a preset set of cache indexes, the index selection module 502 is specifically used for: Based on the data access context, determine the node cluster corresponding to the target client and the throughput, latency, and cost required to read the target data; Select at least one cache index corresponding to the node cluster from the cache index set, and obtain the target cache performance score based on throughput, latency and cost; The target cache index is determined based on at least one cache index that includes a cache performance score greater than the target cache performance score.

[0090] In an optional embodiment, when performing data cache matching on the target data required to execute the target task based on the target cache index to obtain a cache matching result, the cache matching module 503 is specifically used for: Based on the data version number included in the target cache index, the target task is matched against the local cache to obtain the first matching sub-result; the first matching sub-result indicates whether the target local cache that caches the target data was successfully matched. If the first matching sub-result indicates that the target local cache was not successfully matched, then the target task is subjected to distributed cache matching based on the node status of each cache node included in the target cache index to obtain the second matching sub-result; the second matching sub-result indicates whether the target cache node for caching the target data was successfully matched. The cached matching result is obtained based on the first and second matching sub-results.

[0091] In an optional embodiment, after performing local cache matching on the target task based on the data version number included in the target cache index to obtain a first matching sub-result, the cache matching module 503 is further configured to: If the first matching sub-result indicates a successful match with the target local cache, then the first matching sub-result will be used as the cache matching result.

[0092] In an optional embodiment, when selecting the target storage location with the lowest latency required to read the target data from at least one storage location indicated by the cache matching results, the data reading module 504 is specifically configured to: If the cache matching result indicates that the target local cache of the target data has been successfully matched, then the target storage location is determined to be the target local cache; If the cache matching result indicates that the target local cache did not match successfully, but the target cache node successfully matched the target data, then the target storage location is determined to be the target cache node; If the cache matching result indicates that the target local cache and the target cache node were not successfully matched, then the target storage location is determined to be the target object storage included in the target cache index.

[0093] In an optional embodiment, the data reading module 504 is further configured to: If the number of clients reading the target data at the current moment reaches the set threshold, and the target data is cached in the target cache node, then while instructing some of the multiple clients reading the target data to read the target data from the target object storage, the target data will be cached in the idle cache node. After the target data is cached in the idle cache node, another group of clients among multiple clients is instructed to read the target data from both the target cache node and the idle cache node.

[0094] Based on the description of the method and apparatus embodiments above, an exemplary embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the method according to an embodiment of the present invention.

[0095] This application also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.

[0096] This application also provides a computer program product, including a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.

[0097] See Figure 6 The diagram illustrates a structural block diagram of an electronic device 600 that can serve as a server or client in this application, serving as an example of hardware devices applicable to various aspects of this application. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0098] like Figure 6As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0099] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, output unit 607, storage unit 608, and communication unit 609. Input unit 606 can be any type of device capable of inputting information to electronic device 600. Input unit 606 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 607 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 608 may include, but is not limited to, disks and optical discs. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth devices, WiFi devices, worldwide interoperability for microwave access (WiMax) devices, cellular communication devices, and / or the like.

[0100] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various AI computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above.

[0101] For example, in some embodiments, the JuiceFS-oriented data reading method described above can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 600 via ROM 602 and / or communication unit 609. In some embodiments, computing unit 601 can be configured to perform the JuiceFS-oriented data reading method described above by any other suitable means (e.g., by means of firmware).

[0102] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0103] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM) or flash memory, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0104] As used in this application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device, PLD) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0105] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0106] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0107] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0108] Furthermore, it should be understood that the above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of this invention are still within the scope of this application.

Claims

1. A data reading method for JuiceFS, characterized in that, include: In response to a data read request initiated by the target client for the target task, the data read request is parsed to obtain the data access context of the target task; From a preset set of cache indexes, a target cache index that matches the data access context is selected; wherein each cache index in the set of cache indexes is used to indicate the cache status of the data required to perform the corresponding task; Based on the target cache index, the target data required to execute the target task is matched against the data cache to obtain a cache matching result; the cache matching result is used to indicate at least one storage location of the target data. From the at least one storage location indicated by the cache matching result, select the target storage location with the minimum latency required to read the target data, and send the target data read from the target storage location to the target client.

2. The method as described in claim 1, characterized in that, Prior to responding to the data read request initiated by the target client for the target task, the method further includes: The source read cost, task execution priority, and data popularity are obtained for each data stored in multiple object stores; where each source read cost represents the cost required to read the corresponding data from the corresponding object store, and each data popularity represents the reading frequency of the corresponding data within a set time range; Based on the source read cost, task execution priority, and data popularity corresponding to the multiple data, the data storage location corresponding to the multiple data is determined. Cache indexes are set for multiple data storage locations, and multiple cache indexes are saved to the cache index set.

3. The method as described in claim 1, characterized in that, The step of selecting a target cache index that matches the data access context from a preset set of cache indexes includes: Based on the data access context, determine the node cluster corresponding to the target client and the throughput, latency, and cost required to read the target data; Select at least one cache index corresponding to the node cluster from the cache index set, and obtain the target cache performance score based on the throughput, the latency and the cost; The target cache index is determined based on the cache indexes included in the at least one cache index whose cache performance score is greater than the target cache performance score.

4. The method as described in claim 1, characterized in that, The step of performing data cache matching on the target data required to execute the target task based on the target cache index to obtain cache matching results includes: Based on the data version number included in the target cache index, the target task is matched against the local cache to obtain a first matching sub-result; the first matching sub-result indicates whether the target local cache that caches the target data is successfully matched. If the first matching sub-result indicates that the target local cache was not successfully matched, then the target task is subjected to distributed cache matching based on the node status of each cache node included in the target cache index to obtain a second matching sub-result; the second matching sub-result indicates whether the target cache node that caches the target data was successfully matched. The cached matching result is obtained based on the first matching sub-result and the second matching sub-result.

5. The method as described in claim 4, characterized in that, After performing local cache matching on the target task based on the data version number included in the target cache index to obtain the first matching sub-result, the method further includes: If the first matching sub-result indicates a successful match with the target local cache, then the first matching sub-result is taken as the cache matching result.

6. The method as described in claim 1, characterized in that, The step of selecting the target storage location with the minimum latency required to read the target data from the at least one storage location indicated by the cache matching result includes: If the cache matching result indicates a successful match with the target local cache that caches the target data, then the target storage location is determined to be the target local cache; If the cache matching result indicates that the target local cache was not successfully matched but the target cache node that cached the target data was successfully matched, then the target storage location is determined to be the target cache node; If the cache matching result indicates that the target local cache and the target cache node were not successfully matched, then the target storage location is determined to be the target object storage included in the target cache index.

7. The method as described in claim 6, characterized in that, The method further includes: If the number of clients reading the target data at the current moment reaches a set threshold, and the target data is cached in the target cache node, then while instructing some of the multiple clients reading the target data to read the target data from the target object storage, the target data is cached in an idle cache node. After the target data is cached on the idle cache node, another group of clients among the plurality of clients is instructed to read the target data from the target cache node and the idle cache node.

8. A data reading device for JuiceFS, characterized in that, include: The request parsing module is used to respond to a data read request initiated by the target client for the target task, parse the data read request, and obtain the data access context of the target task; The index selection module is used to select a target cache index that matches the data access context from a preset cache index set; wherein each cache index in the cache index set is used to indicate the cache status of the data required to perform the corresponding task; A cache matching module is used to perform data cache matching on the target data required to execute the target task based on the target cache index, and obtain a cache matching result; the cache matching result is used to indicate at least one storage location of the target data. The data reading module is used to filter out the target storage location with the minimum latency required to read the target data from the at least one storage location indicated by the cache matching result, and send the target data read from the target storage location to the target client.

9. An electronic device, characterized in that, include: processor; A memory storing a program; wherein the program includes instructions that, when executed by the processor, cause the processor to perform the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method as described in any one of claims 1-7.

Images