Cache data processing method, related apparatus, and medium

Abstract

A cache data processing method, a related apparatus, and a medium. The method comprises: acquiring a processing request of a processing node for a target block in a target document, the target document being divided into a plurality of blocks in advance, the plurality of blocks comprising the target block, and the processing request comprising a block index of the target block; in a local cache cluster bound to the processing node, determining a target local cache node corresponding to the block index, the local cache cluster comprising a plurality of local cache nodes, and the plurality of local cache nodes comprising the target local cache node; if the target block is stored in the target local cache node, acquiring the target block from the target local cache node; and if the target block is not stored in the target local cache node, acquiring the target block from a data source.

Description

Cached data processing methods, related devices and media

[0001] This application claims priority to Chinese Patent Application No. 2024114454268, filed on October 15, 2024, entitled "Cache data processing method, related apparatus and medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of computer technology, and in particular to a method, apparatus and medium for caching data processing. Background Technology

[0003] Currently, processing nodes in a network environment typically retrieve one or more documents from a data source on a per-document basis for processing. Since retrieving documents from the data source every time is inefficient, a local caching cluster is usually set up within the processing node or at a relevant location. This local caching cluster consists of multiple cache nodes. Each cache node pre-retrieves and stores one or more documents. When a processing node processes a document, it can retrieve the document from the corresponding cache node. If the corresponding cache node fails, the entire document cannot be recovered. Furthermore, even when only a portion of a document's data needs to be processed, the entire document must be retrieved. This results in long document transmission times and low retrieval efficiency. Summary of the Invention

[0004] This disclosure provides a cached data processing method, related apparatus, and medium that can reduce document transmission time and improve document transmission efficiency in document processing.

[0005] According to one aspect of this disclosure, a cached data processing method is provided, performed by an electronic device, the method comprising:

[0006] The processing node obtains a processing request for a target block in the target document, wherein the target document is pre-divided into multiple blocks, the multiple blocks include the target block, and the processing request contains the block index of the target block;

[0007] In the local cache cluster bound to the processing node, the target local cache node corresponding to the block index is determined, wherein the local cache cluster contains multiple local cache nodes, and the multiple local cache nodes include the target local cache node;

[0008] If the target local cache node stores the target block, retrieve the target block from the target local cache node;

[0009] If the target local cache node does not store the target block, the target block is obtained from the data source.

[0010] According to one aspect of this disclosure, a cached data processing apparatus is provided, the apparatus comprising:

[0011] The first acquisition unit is used to acquire the processing request of the processing node for the target block in the target document, wherein the target document is pre-divided into multiple blocks, the multiple blocks include the target block, and the processing request contains the block index of the target block;

[0012] The determining unit is configured to determine the target local cache node corresponding to the block index in a local cache cluster bound to the processing node, wherein the local cache cluster includes multiple local cache nodes, and the multiple local cache nodes include the target local cache node;

[0013] The second acquisition unit is configured to acquire the target block from the target local cache node if the target local cache node stores the target block.

[0014] The third acquisition unit is used to acquire the target block from the data source if the target local cache node does not store the target block.

[0015] According to one aspect of this disclosure, an electronic device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the cached data processing method as described above.

[0016] According to one aspect of this disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program that, when executed by a processor, implements the cached data processing method described above.

[0017] According to one aspect of this disclosure, a computer program product is provided, the computer program product comprising a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the cached data processing method as described above.

[0018] In this embodiment, considering the problem in related technologies where a document cannot be recovered due to a cache node failure after caching it to a cache node, the target document is pre-divided into multiple blocks when caching the target document to the local cache cluster. Caching is performed on a block-by-block basis, enabling fine-grained caching of the target document. This allows different blocks to be cached on different cache nodes. Thus, when a cache node fails, only a block of data within the target document is unrecoverable, not the entire document. Only a small block of document data needs to be recovered, improving the non-volatility of the document data and reducing the difficulty of recovery. Furthermore, after receiving a processing request from the processing node for a target block in the target document, the target local cache node corresponding to the block index can be determined in the local cache cluster bound to the processing node based on the block index contained in the processing request. This allows for quick data querying within the locally deployed local cache cluster of the processing node, improving data query efficiency. Finally, if it is determined that the target local cache node stores the target chunk, the target chunk is directly retrieved from the target local cache node; otherwise, if it is determined that the target local cache node does not store the target chunk, the target chunk is retrieved from the remote data source. This method only transmits the required target chunk when calling cached data, instead of transmitting the entire target document, which can reduce document transmission time in document processing and improve document transmission efficiency.

[0019] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0020] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0021] Figure 1 is a system architecture diagram of a cached data processing method applied according to an embodiment of the present disclosure;

[0022] Figure 2A shows one of the schematic diagrams illustrating the application of the cached data processing method according to an embodiment of the present disclosure in a cached data retrieval scenario;

[0023] Figure 2B shows a second schematic diagram of the application of the cached data processing method according to an embodiment of the present disclosure in a cached data retrieval scenario;

[0024] Figure 2C illustrates a third schematic diagram of the application of the cached data processing method according to an embodiment of the present disclosure in a cached data retrieval scenario;

[0025] Figure 2D illustrates a fourth schematic diagram of the application of the cached data processing method according to an embodiment of the present disclosure in a cached data retrieval scenario;

[0026] Figure 3 is a flowchart of a cached data processing method according to an embodiment of the present disclosure;

[0027] Figure 4 is a flowchart of assigning processing tasks to processing nodes according to an embodiment of the present disclosure;

[0028] Figure 5 is a flowchart of determining a second score based on the current node state according to an embodiment of the present disclosure;

[0029] Figure 6A is one of the schematic diagrams illustrating the implementation process of determining the total node score of a candidate processing node according to an embodiment of the present disclosure;

[0030] Figure 6B is a second schematic diagram illustrating the implementation process of determining the total node score of candidate processing nodes according to an embodiment of the present disclosure;

[0031] Figure 7A is one of the schematic diagrams illustrating the implementation process of calling cached data when performing a processing task according to an embodiment of the present disclosure;

[0032] Figure 7B is a second schematic diagram illustrating the implementation process of calling cached data when performing a processing task according to an embodiment of the present disclosure;

[0033] Figure 8 is a flowchart of pre-storing frequently used data in a local cache cluster according to an embodiment of the present disclosure;

[0034] Figure 9 is a schematic diagram illustrating the data filtering process when frequently used data is pre-stored in a local cache cluster according to an embodiment of the present disclosure;

[0035] Figure 10 is a flowchart of caching target documents in chunks to a local cache cluster according to an embodiment of the present disclosure;

[0036] Figure 11A is one of the schematic diagrams illustrating the process of dividing a target document into multiple blocks according to an embodiment of the present disclosure;

[0037] Figure 11B is a second schematic diagram illustrating the process of dividing a target document into multiple blocks according to an embodiment of the present disclosure;

[0038] Figure 12 is a schematic diagram of the implementation process of storing blocks into the local cache node corresponding to the block index according to an embodiment of the present disclosure;

[0039] Figure 13 is a flowchart of determining the block index range of each local cache node according to an embodiment of the present disclosure;

[0040] Figure 14A is one of the schematic diagrams illustrating the implementation process of storing blocks into the local cache node corresponding to the block index according to an embodiment of the present disclosure;

[0041] Figure 14B is a second schematic diagram illustrating the implementation process of storing blocks into the local cache node corresponding to the block index according to an embodiment of the present disclosure.

[0042] Figure 15 is a flowchart of storing blocks into the local cache node corresponding to the block index according to an embodiment of the present disclosure;

[0043] Figure 16 is a schematic diagram illustrating the implementation process of storing blocks into the local cache node corresponding to the block index according to an embodiment of the present disclosure;

[0044] Figure 17 is a flowchart of cached data processing when a node exits, according to an embodiment of the present disclosure;

[0045] Figure 18 is a flowchart of cache data processing when a node is added, according to an embodiment of the present disclosure;

[0046] Figure 19 is a flowchart of cached data processing in the event of a node failure according to an embodiment of the present disclosure;

[0047] Figure 20 is a flowchart of evicting cached data according to an embodiment of the present disclosure;

[0048] Figure 21 is a flowchart of evicting cached data according to another embodiment of the present disclosure;

[0049] Figure 22A is one of the schematic diagrams illustrating the implementation process of evicting cached data according to an embodiment of the present disclosure;

[0050] Figure 22B is a second schematic diagram illustrating the implementation process of evicting cached data according to an embodiment of the present disclosure;

[0051] Figure 23 is a block diagram of a cached data processing apparatus according to an embodiment of the present disclosure;

[0052] Figure 24 is a terminal structure diagram of a cached data processing method according to an embodiment of the present disclosure;

[0053] Figure 25 is a server structure diagram of a cached data processing method according to an embodiment of the present disclosure. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0055] The system architecture and scenarios in which this disclosure is applied are described below.

[0056] Figure 1 is a system architecture diagram of the cached data processing method according to an embodiment of the present disclosure. It includes an object terminal 140, an Internet 130, a gateway 120, a cluster computing server 150, a local cache cluster system 110, and a remote data source 160, etc.

[0057] The object terminal 140 can take various forms, including desktop computers, laptops, PDAs (Personal Digital Assistants), mobile phones, in-vehicle terminals, home theater terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. The object terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data. The object terminal 140 includes a data processing platform, which supports the object in initiating processing tasks for various types of data.

[0058] Cluster computing server 150 refers to a computer system that can provide certain services to object terminal 140. Compared with ordinary object terminal 140, cluster computing server 150 has higher requirements in terms of stability, security, and performance. Cluster computing server 150 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Cluster computing server 150 includes various types of services, and the implementation of each service of cluster computing server 150 is often associated with some intermediate databases or storage media. Cluster computing server 150 is used to receive processing tasks for various data initiated by objects and call cluster nodes to execute the processing tasks; local caching cluster system 110 is used to obtain various data from remote data source 160 and cache the obtained data at a small granularity; remote data source 160 is used to store various data.

[0059] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from target terminal 140 to cluster computing server 150 or local caching cluster system 110 must be forwarded to the corresponding cluster computing server 150 or local caching cluster system 110 via gateway 120. Similarly, messages sent from cluster computing server 150 or local caching cluster system 110 to target terminal 140 must also be forwarded to the corresponding target terminal 140 via gateway 120.

[0060] The embodiments disclosed herein can be applied to various scenarios, such as the cached data retrieval scenarios shown in Figures 2A-2D.

[0061] As shown in Figure 2A, when an object needs to access a portion of document data to perform a computational task, it triggers a cached data retrieval process on the data processing platform. At this time, the data processing platform page displays a prompt field, "Please enter the document information associated with the data processing task," and an input area for entering the document information. Based on this, the object enters "Document Name: Document K, the index of the target document data to be retrieved is xxxxx, xxx" in the input area and clicks the "Confirm" button.

[0062] As shown in Figure 2B, after the object clicks the "Confirm" button, a prompt field "The current number of tasks for each task node in the task processing cluster is as follows" will be displayed on the page, showing that the current number of tasks for task node 1 is 12, the current number of tasks for task node 2 is 4, the current number of tasks for task node 3 is 5, and the current number of tasks for task node 4 is 8. In addition, a prompt field "Based on this, the current data processing task is assigned to task node 2, and task node 2 will call the target document data to execute the data processing task" will also be displayed on the page, indicating which task node is specifically executing the data processing task.

[0063] As shown in Figure 2C, after the document data of document K has been cached at a small granularity, a prompt window will be displayed on the page. The prompt window displays the prompt field "The target document block data of document K has been found in the local cache cluster. The target document block data is being retrieved from the local cache cluster. Please wait patiently...".

[0064] As shown in Figure 2D, when the document data of document K is not cached in the local cache cluster, a prompt window will be displayed on the page. The prompt window displays the message "The target document block data of document K was not found in the local cache cluster. The document data of document K is being retrieved from the remote data source. Please wait patiently...".

[0065] The embodiments of this disclosure are described in general below.

[0066] According to one embodiment of this disclosure, a method for processing cached data is provided.

[0067] This cached data processing method is generally applied in business scenarios where cached data needs to be invoked to perform various tasks, such as the cached data invocation scenarios shown in Figures 2A-2D. This disclosure provides a scheme for implementing cached data invocation based on local cache nodes of a local cache cluster, which can reduce document transmission time and improve document transmission efficiency. The cached data processing method of this disclosure can be executed by an electronic device, such as by a server corresponding to the local cache cluster.

[0068] As shown in Figure 3, a cached data processing method according to an embodiment of this disclosure may include:

[0069] Step 310: Obtain the processing request from the processing node for the target block in the target document;

[0070] Step 320: In the local cache cluster bound to the processing node, determine the target local cache node corresponding to the block index;

[0071] Step 330: If the target local cache node stores the target block, retrieve the target block from the target local cache node;

[0072] Step 340: If the target local cache does not store the target block, retrieve the target block from the data source.

[0073] Steps 310-340 are described in detail below.

[0074] In step 310, the processing node obtains the processing request for the target block in the target document.

[0075] A processing node refers to a cluster node belonging to a computing cluster and used to process various tasks.

[0076] A target document refers to a document file associated with the computational task that the processing node is to process. A portion of the document data in the target document can help the computational task to execute normally; this portion of the document data can be called a target chunk.

[0077] The target document is pre-divided into multiple blocks. For example, pages in the target document can be pre-divided into multiple blocks along the row dimension, or along the column dimension, or even along both the row and column dimensions. This embodiment of the disclosure does not limit the method of dividing the target document into multiple blocks. The multiple blocks include the target block.

[0078] A processing request refers to a request initiated by a processing node when it wants to perform a processing task (or a computational task related to the target document) on the target document. This processing request is used to request some cached data from the local cache cluster so that the processing task can be executed normally.

[0079] The processing request includes the block index of the target block.

[0080] The block index is used to identify the target block, and different blocks have different block indexes.

[0081] In step 320, the target local cache node corresponding to the block index is determined in the local cache cluster bound to the processing node.

[0082] A local cache cluster refers to a collection of cache services deployed locally, close to the processing node. A local cache cluster consists of multiple local cache nodes that work together to provide efficient data storage and retrieval services. The local cache cluster setup in this embodiment effectively improves data access speed and reduces the pressure on the backend database.

[0083] Among them, multiple local cache nodes contain the target local cache node.

[0084] A local cache node refers to a cluster node in a local cache cluster that has a certain amount of cache space and is capable of storing various types of data.

[0085] The target local cache node refers to the local cache node where the target block is cached.

[0086] In this specific implementation, since there is often an inherent mapping relationship between the cached data cached on the local cache nodes and the local cache nodes themselves in the local cache cluster, after obtaining the block index of the target block, the local cache node corresponding to the block index of the target block can be found in the local cache cluster bound to the processing node, based on the inherent mapping relationship between the cached data cached on the local cache nodes and the local cache nodes, as the target local cache node.

[0087] For example, a mapping relationship can be established between the block index of the cached blocks in each local cache node and the node number of the local cache node. Based on the block index, the local cache node corresponding to the node number and the block index of the target block can be used as the target local cache node.

[0088] For example, you can set the index range of each local cache node, and use the local cache node corresponding to the index range to which the block index of the target block belongs as the target local cache node.

[0089] In step 330, if the target local cache node stores the target block, the target block is retrieved from the target local cache node.

[0090] In the specific implementation of this embodiment, if the target local cache node stores the target block, it indicates that the target block has been cached on the target local cache node. Therefore, the target block can be directly obtained from the target local cache node.

[0091] It should be noted that when a target block is invoked, certain specific columns can be accessed, which is beneficial for efficient reading of cached data.

[0092] In step 340, if the target local cache does not store the target block, the target block is retrieved from the data source.

[0093] Data sources refer to remote servers or cloud services that store data.

[0094] In this specific implementation, if the target local cache does not store the target chunk, it indicates that the target document's target chunk has not been cached in the local cache cluster. Based on this, with authorization, the entire document data of the target document is obtained from the data source, and the target chunk is determined within the entire document data. Then, after caching the target document in multiple chunks in the local cache cluster, the target chunks are fed back to the processing node.

[0095] Through steps 310-340 above, this embodiment of the present disclosure considers the problem in related technologies where a document cannot be recovered if a cache node fails after caching it. By pre-dividing the target document into multiple blocks and caching them in units of blocks when caching the target document to a local cache cluster, fine-grained caching of the target document can be achieved. This allows each block to potentially be cached on different cache nodes. Thus, when a cache node fails, only a block of data within the target document is unrecoverable, not the entire document. Only a small block of document data needs to be recovered, improving the non-volatility of document data and reducing the difficulty of document data recovery. Furthermore, after receiving a processing request from the processing node for a target block in the target document, the target local cache node corresponding to the block index can be determined in the local cache cluster bound to the processing node based on the block index of the target block contained in the processing request. This allows for quick data querying within the locally deployed local cache cluster of the processing node, improving data query efficiency. Finally, if it is determined that the target local cache node stores the target chunk, the target chunk is directly retrieved from the target local cache node; otherwise, if it is determined that the target local cache node does not store the target chunk, the target chunk is retrieved from the remote data source. This method only transmits the required target chunk when calling cached data, instead of transmitting the entire target document, which can reduce document transmission time in document processing and improve document transmission efficiency.

[0096] The above is a general description of steps 310-340. The following will provide a detailed description of the specific implementation of steps 310-340.

[0097] Step 310 will be described in detail below.

[0098] In step 310, the processing node obtains the processing request for the target block in the target document, wherein the target document is pre-divided into multiple blocks, the multiple blocks include the target block, and the processing request contains the block index of the target block.

[0099] Referring to Figure 4, in one embodiment, the processing request for the target chunk is generated by the processing node based on the received processing task for the target document; the processing task for the target document is allocated to the processing node by the task scheduling node in the task cluster through the following process:

[0100] Step 410: Obtain the current number of tasks and the current node status of each candidate processing node in the task cluster;

[0101] Step 420: For each candidate processing node, determine the first score based on the current number of tasks;

[0102] Step 430: Determine the second score based on the current node state;

[0103] Step 440: Based on the first score and the second score, determine the total node score of the candidate processing node;

[0104] Step 450: Based on the total node score, determine the processing node from multiple candidate processing nodes to perform the processing task for the target document, and assign the processing task to the processing node.

[0105] Steps 410-450 are described in detail below.

[0106] A task cluster refers to a cluster that can efficiently process large amounts of data. A task cluster typically contains a task scheduling node and multiple candidate processing nodes.

[0107] A task scheduling node is a node in a task cluster that is responsible for coordinating task scheduling and data acquisition among various candidate processing nodes. The node functions of a task scheduling node include allocating tasks to candidate processing nodes so that the available resources of the candidate processing nodes can be used to execute the tasks.

[0108] Candidate processing nodes refer to cluster nodes that execute tasks assigned by the task scheduling node.

[0109] A processing task targeting a target document refers to a data processing task performed on a subset of document data (target chunks) within the target document. These processing tasks are often generated based on user requests in different business scenarios.

[0110] In step 410, the current number of tasks of the candidate processing node is used to indicate the total number of tasks that the candidate processing node is currently processing.

[0111] The current node status of a candidate processing node is used to indicate the processing delay and resource status of the candidate processing node at the current time.

[0112] In this specific implementation, the working status of each candidate processing node in the task cluster is recorded in real time by the cluster computing server in the server background. Based on this, with authorization, the task scheduling node can obtain the current number of tasks and the current node status of each candidate processing node from the background logs of the cluster computing server.

[0113] In step 420, the first score is used to statistically indicate the saturation level of the tasks being processed by the candidate processing node.

[0114] In this specific implementation, a function calculation method can be used to calculate the first score of the candidate processing node based on the current number of tasks. Specifically, firstly, a preset function is invoked. This preset function is a functional equation indicating how the first score changes with the current number of tasks. The preset function is a decreasing function with the current number of tasks as the independent variable and the first score as the dependent variable. The larger the current number of tasks, the smaller the first score. Next, the current number of tasks of the candidate processing node is input into the preset function, and the output of the preset function is used as the first score of the candidate processing node.

[0115] In step 430, the second score is used to statistically indicate the quality of the candidate processing node's node state at the current time.

[0116] To save space, the specific process of determining the second score based on the current node state in this embodiment will be described in detail below, and will not be repeated here.

[0117] In step 440, the total node score is used to statistically indicate how well the candidate processing node is suited for processing the target file. Specifically, a higher total node score indicates that the candidate processing node is more suitable as a processing node for processing the target file.

[0118] In the specific implementation of this embodiment, firstly, a first weight and a second weight are determined. The first weight indicates the influence of the current number of tasks of the candidate processing node on node selection; the second weight indicates the influence of the current node state of the candidate processing node on node selection. The sum of the first weight and the second weight is 1. Next, the first weight and the first score are multiplied to obtain a first product result; the second weight and the second score are multiplied to obtain a second product result; the first product result and the second product result are added together to obtain the total node score of the candidate processing node.

[0119] In step 450, based on the total node score, a processing node is determined from multiple candidate processing nodes to perform the processing task for the target document, and the processing task is assigned to the target processing node.

[0120] In this specific implementation, when determining the processing node to execute the processing task for the target document from multiple candidate processing nodes, a higher total node score indicates a more balanced number of current tasks and a more balanced current node state at the current time, thus better meeting the conditions for assigning processing tasks. Therefore, the candidate processing node with the highest total node score can be selected first as the processing node to execute the processing task for the target document, thereby improving the task execution efficiency. Then, the processing task is assigned to this processing node.

[0121] In another embodiment, when determining the processing node for performing the processing task for the target document from multiple candidate processing nodes, one can first filter out candidate processing nodes whose total score is greater than a preset threshold as intermediate processing nodes. Then, one of the intermediate processing nodes is randomly selected as the processing node for performing the processing task for the target document, and the processing task is assigned to that node. This method improves the flexibility and freedom of node selection when choosing a processing node that satisfies both the current number of tasks and the current node status as task assignment criteria.

[0122] The advantage of this embodiment is that when the task scheduling node allocates processing tasks, it takes into account both the current number of tasks and the current node status of each candidate processing node. It also quantifies the impact of the current number of tasks and the current node status of each candidate processing node on node selection from a statistical perspective, calculates the total node score of each candidate processing node, and then uses the total node score to perform node selection. This helps to select candidate processing nodes with fewer current tasks and better current node status to perform processing tasks on the target document, thereby improving the rationality of task allocation and improving the efficiency and reliability of task execution.

[0123] In this embodiment of the disclosure, the current node status of the candidate processing node includes the node latency level and the node resource status.

[0124] Node latency is used to indicate the time or speed at which a candidate processing node processes a single task.

[0125] Node resource status is used to indicate how much computing resources a candidate processing node is allowed to access at the current time.

[0126] Referring to Figure 5, in one embodiment, step 430 specifically includes, but is not limited to, the following steps 510-530:

[0127] Step 510: Determine the first sub-score based on the node latency.

[0128] Step 520: Determine the second sub-score based on the node resource status;

[0129] Step 530: Determine the second score of the candidate processing node based on the weighted sum of the first sub-score and the second sub-score.

[0130] Steps 510-530 are described in detail below.

[0131] In step 510, the first sub-score is used to quantitatively evaluate the node latency of candidate processing nodes from a statistical perspective. Specifically, the greater the node latency, the smaller the first sub-score.

[0132] In the specific implementation of this embodiment, the process of step 510 is similar to that of step 420 described above. To save space, it will not be described again.

[0133] In step 520, the second sub-score is used to quantitatively evaluate the node resource status of the candidate processing node from a statistical perspective. Specifically, the more computing resources available on a candidate processing node, the better its node resource status, and the larger the second sub-score.

[0134] In the specific implementation of this embodiment, the process of step 520 is similar to that of step 420 described above. To save space, it will not be described again.

[0135] In step 530, the second score of the candidate processing node is determined based on the weighted sum of the first sub-score and the second sub-score.

[0136] In the specific implementation of this embodiment, the process of step 530 is similar to that of step 440 described above. To save space, it will not be described again.

[0137] Figure 6A illustrates the calculation of the second score for each candidate processing node. Specifically, for candidate processing node 1, with a node latency of 20ms and a node resource status (available computing resources) of 100, the first sub-score is 80 and the second sub-score is 80; for candidate processing node 2, with a node latency of 30ms and a node resource status (available computing resources) of 150, the first sub-score is 70 and the second sub-score is 85; for candidate processing node 3, with a node latency of 25ms and a node resource status (available computing resources) of 80, the first sub-score is 75 and the second sub-score is 78; and for candidate processing node 4, with a node latency of 40ms and a node resource status (available computing resources) of 200, the first sub-score is 60 and the second sub-score is 90. Based on this, when the weights of the first and second sub-scores are both 0.5, the second score of candidate processing node 1 is 80, the second score of candidate processing node 2 is 77.5, the second score of candidate processing node 3 is 76.5, and the second score of candidate processing node 4 is 75.

[0138] Figure 6B illustrates the calculation of the total score for each candidate processing node. Specifically, candidate processing node 1 has 2 current tasks, so its first score is 90; candidate processing node 2 has 4 current tasks, so its first score is 80; candidate processing node 3 has 5 current tasks, so its first score is 75; and candidate processing node 4 has 8 current tasks, so its first score is 60. Based on this, when both the first and second weights are 0.5, the total score for candidate processing node 1 is 85, for candidate processing node 2 it is 78.75, for candidate processing node 3 it is 75.75, and for candidate processing node 4 it is 67.5. Therefore, candidate processing node 1 is selected as the processing node for the task targeting the document.

[0139] The advantage of this embodiment is that, when calculating the second score for each candidate processing node, it considers analyzing the current node state from multiple dimensions, such as node latency and node resource status. Therefore, by statistically quantifying the impact of node latency and node resource status on node state analysis based on the candidate processing node's latency and resource status, and calculating the second score for each candidate processing node, a comprehensive consideration of the factors influencing node state can be achieved, thereby improving the accuracy of score calculation.

[0140] The following provides an example description of steps 320-340.

[0141] In steps 320-340, the target local cache node corresponding to the block index is determined in the local cache cluster bound to the processing node. If the target local cache node stores the target block, the target block is obtained from the target local cache node; if the target local cache does not store the target block, the target block is obtained from the data source.

[0142] Figure 7A illustrates a simplified flowchart of cached data processing. Specifically, when the data processing engine or the Spark Engine needs to access document data during various computational tasks, it first retrieves the relevant document data from the local fine-grained cache. Only when the relevant document data is not present in the local fine-grained cache does it obtain the document data from the remote data source, cache it in the local cache, and then transmit it to the data processing engine or the Spark Engine.

[0143] Figure 7B illustrates the specific implementation process of invoking cached data during task execution. Specifically, when a processing task for a target document needs to be executed, the cluster computing system includes a task scheduling node (Spark Driver) and several candidate processing nodes (Spark Executors). The task scheduling node selects candidate processing node 1 from among candidate processing node 1, candidate processing node 2, and candidate processing node 3 as the processing node to execute the processing task for the target document, and assigns the processing task to it, thus achieving distributed scheduling of tasks. Next, when candidate processing node 1 executes the processing task for the target document, it first requests cached data related to the processing task (e.g., the target block of the target document) from the local cache cluster (Data United Cache). Further, based on the request sent by the target processing node (candidate processing node 1), it determines whether a local cache node in the local cache cluster has been hit. If a local cache node corresponding to the block index of the target block exists in the local cache cluster, it is determined that a local cache node in the local cache cluster has been hit, and the cached data is retrieved from the cache pool of the local cache cluster and fed back to candidate processing node 1. If no local cache node corresponding to the block index of the target block exists in the local cache cluster, it is determined that a local cache node in the local cache cluster has not been hit, and the Triger Data loading process is triggered to automatically load the data into the local cache cluster (Cache data loading plugin). Data is retrieved from a remote data source, where the data is stored in columnar storage format Parquet or columnar storage format ORC. Further, when data related to the processing task is found in the remote data source, the data is uploaded to the local cache cluster, and after the data is saved to the cache pool of the local cache cluster, the cached data is retrieved and fed back to candidate processing node 1. In addition, a corresponding data update mechanism can be set in the cache pool of the local cache cluster to evict and update various cached data. The local cache cluster can be a self-built distributed cluster.

[0144] The advantage of this embodiment is that, during task execution, the task scheduling node uniformly schedules and allocates each task, which helps improve the rationality of task allocation. Furthermore, after a candidate processing node receives an allocated task, it first checks if a local cache node is matched in the local cache cluster. If a local cache node is matched, cached data can be directly retrieved from the local cache node. Only if a local cache node is not matched in the local cache cluster will document data be retrieved from a remote data source. After caching the document data, the cached data required by the task is then fed back to the candidate processing node. This method reduces document transmission time during document processing and improves document transmission efficiency. In addition, since the local cache cluster is bound to the candidate processing node, storing cached data near the computing node (candidate processing node) achieves cache locality, allowing access to cached data to be implemented in the local cache. This helps reduce network transmission distance and time, improving data access speed.

[0145] The process of pre-storing frequently used data in a local cache cluster according to one embodiment of the present disclosure is described in detail below.

[0146] Referring to Figure 8, in one embodiment, before obtaining the processing request from the processing node for the target chunk in the target document, the cached data processing method further includes, but is not limited to, the following steps 810-820:

[0147] Step 810: Determine the data retrieval frequency and data size for each of the multiple reference data sets;

[0148] Step 820: Based on the data call frequency and data size, select candidate data from multiple reference data and cache the candidate data in the local cache cluster.

[0149] Steps 810-820 are described in detail below.

[0150] In step 810, the data retrieval frequency and data size of each of the multiple reference data are determined.

[0151] Reference data refers to data that can be provided by various data sources, is frequently used, and is not often modified.

[0152] Data retrieval frequency is used to indicate the number of times reference data will be retrieved within a fixed time period.

[0153] Data size is used to indicate the amount of data that the reference data has. For example, data size can refer to the number of bytes in the reference data.

[0154] In the specific implementation of this embodiment, since the data call frequency and data size of each reference data are often recorded in the server corresponding to the data source where the reference data is located, with authorization, the data call frequency and data size of the reference data can be retrieved from the background log of the server corresponding to the data source where the reference data is located.

[0155] In step 820, candidate data is determined from multiple reference data based on data call frequency and data size, and the candidate data is cached in the local cache cluster.

[0156] Alternate data refers to the specific data that is ultimately selected to be cached in the local cache cluster before the task is executed by the calling processing node.

[0157] In a specific implementation of this embodiment, step 820 may include, but is not limited to, the following steps:

[0158] The third score is determined based on the frequency of data access.

[0159] The fourth score is determined based on the size of the data.

[0160] The data scores for the reference data are determined based on the third and fourth scores;

[0161] Based on the data scores, alternative data are determined from multiple reference data, and the alternative data is cached in the local cache cluster.

[0162] Specifically, the process of determining the third score based on data call frequency and the process of determining the fourth score based on data size are similar to step 420 above. The process of determining the data score of the reference data based on the third and fourth scores is similar to step 440 above. The process of determining the candidate data from multiple reference data based on the data score is similar to step 450 above. To save space, these will not be elaborated further.

[0163] It should be noted that the specific process of caching candidate data into the local cache cluster in this embodiment is basically the same as the specific process of caching each block of the target document into the local cache cluster. Since the specific process of caching each block of the target document into the local cache cluster will be described in detail below, it will not be repeated here.

[0164] Figure 9 illustrates the specific calculation of data scores for each reference data point. Specifically, for reference data 1, with a data access frequency of 20, a data size of 100 bytes, and originating from data source 1, the third score is 60 and the fourth score is 80. For reference data 2, with a data access frequency of 30, a data size of 150 bytes, and originating from data source 1, the third score is 65 and the fourth score is 85. For reference data 3, with a data access frequency of 25, a data size of 80 bytes, and originating from data source 2, the third score is 70 and the fourth score is 92. For reference data 4, with a data access frequency of 40, a data size of 200 bytes, and originating from data source 3, the third score is 80 and the fourth score is 80. Based on this, when the weights of the third and fourth scores are both 0.5, the data score for reference data 1 is 75, the data score for reference data 2 is 75, the data score for reference data 3 is 81, and the data score for reference data 4 is 80. Based on this, reference data 3 and reference data 4, which have higher data scores, are used as alternative data and are pre-stored in the local cache cluster.

[0165] The advantage of this embodiment is that it sets up an active caching mechanism in the local cache cluster, enabling the local cache cluster to proactively pre-store frequently used data from various data sources after its creation. This proactive data preheating avoids waiting until a data retrieval request is received before retrieving and caching data, allowing the caching of frequently used data to be performed in advance, thereby improving the efficiency of cache data retrieval and task execution. This active caching method can preheat and adjust cache content, which is beneficial for adapting to different load conditions, improving cache hit rate, and reducing performance issues caused by dynamic load changes.

[0166] The following is a detailed description of an embodiment of the present disclosure of caching target documents to a local cache cluster.

[0167] Referring to Figure 10, in one embodiment, before obtaining the processing request from the processing node for the target block in the target document, the cached data processing method further includes, but is not limited to, the following steps 1010-1030:

[0168] Step 1010: Divide the pages in the target document into multiple blocks along the dimensions of rows and columns;

[0169] Step 1020: Determine the block index based on the block's digest value;

[0170] Step 1030: Store the blocks in the local cache node corresponding to the block index.

[0171] Steps 1010-1030 are described in detail below.

[0172] In step 1010, the pages in the target document are divided into multiple blocks in terms of rows and columns.

[0173] In this specific implementation, firstly, the pages in the target document are divided into multiple row groups along the row dimension, where each row group contains one or more rows of data. Next, each row group is divided into multiple column blocks along the column dimension, with each column block serving as a separate block, resulting in multiple blocks. Each block includes one or more columns of data. This allows the data of a file (e.g., a document) to be divided into smaller units, and a high-efficiency data structure is used for cache management at the small unit (e.g., block) level, achieving fine-grained caching.

[0174] Figure 11A illustrates how a document is divided into multiple chunks along the dimensions of rows and columns. Specifically, a document contains multiple rows of data, with the footer at the end primarily containing metadata describing the document's structure, pages, and other detailed information. When dividing the document, it is first divided into multiple row groups, each containing multiple rows of data. Next, each row group is further divided into multiple column chunks, each containing the document content of a specific column from a specific row.

[0175] Figure 11B illustrates how a document is divided into multiple blocks along the dimensions of rows and columns. Specifically, when dividing a document along the dimensions of rows and columns, the document is divided into 20 blocks of equal size. When divided along the row dimension, the document is divided into 5 row groups. When each row group is divided along the column dimension, each row group is further divided into 4 column blocks, resulting in 20 blocks.

[0176] In steps 1020-1030, the block index is determined based on the block's digest value. The block is then stored in the local cache node corresponding to the block index.

[0177] The digest value of a block refers to the result obtained by performing a digest operation on the specific data in the block using a predetermined digest algorithm. The digest value of a block is often a random string of fixed length.

[0178] To save space, the specific process of determining the block index and caching the blocks to the local cache node in this embodiment will be described in detail below, and will not be repeated here.

[0179] The advantage of this embodiment is that, when caching the target document, it does not cache the entire target document as a whole, but rather divides the target document into multiple blocks along the row and column dimensions, and caches them on a block-by-block basis, transforming coarse-grained caching into fine-grained caching. Furthermore, when caching a single block, the block index is determined based on the block's digest value, and the block is stored in the corresponding local cache node according to the mapping relationship between the block index and each local cache node, which improves the standardization and rationality of block caching.

[0180] It should be noted that when cached data is cached in a local cache cluster, a multi-level caching approach is adopted, caching data at different levels to reduce the number of accesses to the underlying storage system (remote data source), thereby reducing latency and improving throughput. Specifically, the multi-level caching in this embodiment includes in-process caching, node caching, and distributed caching. In-process caching is used for frequently accessed cached data, caching it using the memory space of a single process. Node caching caches data in the memory of a single local cache node, making the cached data shared by all processes corresponding to the local cache node, improving the utilization rate of cached data. Distributed caching shares cached data among multiple local cache nodes in the local cache cluster, performing distributed caching based on the cached data's summary information.

[0181] In one embodiment, the specific process of determining the block index based on the block's digest value includes:

[0182] Determine the remainder of the summary value modulo the first number, and use it as the block index.

[0183] The first number is the number of local cache nodes in the local cache cluster.

[0184] Specifically, firstly, a predefined digest algorithm is used to perform a digest operation on the specific data in the blocks, obtaining the digest value of each block. Next, the number of local cache nodes in the local cache cluster is counted to obtain a first count. Further, the digest value is modulo the first count to obtain a remainder, which is then used as the block index. This method simplifies the representation of the block index.

[0185] In this embodiment, the specific process of storing the blocks into the local cache node corresponding to the block index includes:

[0186] The blocks are stored in local cache nodes with the same sequence number as the block index.

[0187] Specifically, first, the node sequence number assigned to each local cache node in the local cache cluster is obtained. The node sequence number can be set according to the total number of local cache nodes in the local cache cluster. Next, based on the node sequence number, the local cache node whose node sequence number matches the block index is found, and the block is cached in the found local cache node.

[0188] Figure 12 illustrates a specific example of storing blocks in the local cache nodes corresponding to the block index. Specifically, each local cache node is assigned a node number; for example, the node number of local cache node 1 is set to 1, the node number of local cache node 2 is set to 2, and so on. Based on this, when the remainder of the summary value of a block of the target document modulo the first number of all local cache nodes is 13, the block is cached on local cache node 13 with node number 13.

[0189] The advantage of this embodiment is that when determining the block index of the local cache node, the digest value of the block and the total number of local cache nodes are calculated so that the block index can be represented by a single value, which simplifies the representation of the block index. At the same time, storing the block in the local cache node with the same sequence number as the block index can achieve reasonable caching of the block and make the cached data of each local cache node more balanced.

[0190] In another embodiment, the specific process of determining the block index based on the block's digest value includes:

[0191] The summary value is determined as the block index.

[0192] Specifically, after performing a digest operation on the specific data in the blocks to obtain the digest value, the digest value is directly used as the block index, thereby simplifying the process of determining the block index.

[0193] In this embodiment, the local cache node has a block index range, and the specific process of storing blocks into the local cache node corresponding to the block index includes:

[0194] The blocks are stored in the local cache node corresponding to the block index range to which the block index belongs.

[0195] The block index range is used to indicate the specific range of values ​​for the digest value corresponding to the cached data that the local cache node is allowed to cache.

[0196] Specifically, first, the block index range to which the block index belongs is determined by comparing the block index range of each local cache node with the block index itself. Then, the block is stored in the local cache node corresponding to the block index range to which the block index belongs.

[0197] The advantage of this embodiment is that, when determining the block index of a local cache node, directly using the block digest value as the block index improves the efficiency of block index determination. Simultaneously, pre-allocating a block index range for each local cache node and storing the blocks in the local cache node corresponding to the block index range to which the block index belongs improves the accuracy of block caching.

[0198] Referring to Figure 13, in one embodiment, the specific process of determining the block index range of a local cache node may include, but is not limited to, the following steps 1310-1340:

[0199] Step 1310: Obtain the preset summary data body;

[0200] Step 1320: Determine the total number of cache nodes in the local cache cluster;

[0201] Step 1330: Based on the total number of cache nodes, divide the digest data body into equal parts to obtain multiple digest data segments;

[0202] Step 1340: Match multiple local cache nodes with multiple digest data segments so that each local cache node corresponds to a digest data segment, and use the digest data segment as the block index range corresponding to the local cache node.

[0203] Steps 1310-1340 are described in detail below.

[0204] In step 1310, a preset summary data body is obtained.

[0205] The summary data body is a ring-shaped space with multiple slots, each slot corresponding to a summary value.

[0206] In this specific implementation, firstly, a series of digest values ​​are determined. Then, this series of digest values ​​is mapped to a ring space to obtain the digest data volume.

[0207] For example, a hash space can be mapped to a virtual ring, and the virtual ring can be used as the digest data body, where the hash space takes values ​​from 0 to (2^32-1).

[0208] In step 1320, the total number of cache nodes in the local cache cluster is determined.

[0209] The total number of cache nodes indicates the total number of local cache nodes in the local cache cluster.

[0210] In the specific implementation of this embodiment, the summation function or similar methods can be used to count the number of local cache nodes in the local cache cluster to obtain the total number of cache nodes.

[0211] In step 1330, based on the total number of cache nodes, the digest data body is divided into multiple digest data segments.

[0212] A summary data segment is a data segment consisting of multiple consecutive summary values.

[0213] In this specific implementation, the digest data body is divided equally according to the total number of cache nodes to obtain a digest data segment of the total number of cache nodes.

[0214] In step 1340, multiple local cache nodes are matched with multiple digest data segments so that each local cache node corresponds to a digest data segment, and the digest data segment is used as the block index range corresponding to the local cache node.

[0215] In this specific implementation, when matching multiple local cache nodes with multiple digest data segments, the local cache nodes and digest data segments are first randomly matched so that each local cache node corresponds to one digest data segment, and different local cache nodes are matched with different digest data segments. Then, for each local cache node, the matched digest data segment is used as the block index range corresponding to that local cache node.

[0216] Figure 14A illustrates the specific steps for setting block index intervals for each local cache node. Specifically, for a circular digest data body, when there are four local cache nodes in the local cache cluster, the circular digest data body is divided into four equal parts, resulting in four digest data segments of the same size. Further, the digest data segment in the upper left corner is designated as block index interval 1 for local cache node 1; the digest data segment in the lower left corner is designated as block index interval 3 for local cache node 3; the digest data segment in the upper right corner is designated as block index interval 2 for local cache node 2; and the digest data segment in the lower right corner is designated as block index interval 4 for local cache node 4.

[0217] Figure 14B illustrates a specific example of storing blocks in the local cache node corresponding to the block index. Specifically, for a single block whose block index is in block index range 1, the mapping position of this block in the digest data body is within the range of block index range 1. Therefore, this single block is cached on local cache node 1. Similarly, for a single block whose block index is in block index range 3, the mapping position of this block in the digest data body is within the range of block index range 3. Therefore, this single block is cached on local cache node 3.

[0218] The advantage of this embodiment is that by dividing a circular digest data body equally according to the total number of local cache nodes in the local cache cluster, multiple digest data segments with equal data volume are obtained. This enables the uniform distribution of digest data segments, ensuring that the number of digest values ​​contained in the block index interval of each local cache node is equal. This improves the rationality of determining the block index interval, which is conducive to realizing distributed caching of cached data and improving the load balancing of each local cache node.

[0219] Referring to Figure 15, in another embodiment, the specific process of determining the block index based on the block's digest value includes:

[0220] The summary value is determined as the block index.

[0221] In this embodiment, step 1030 specifically includes, but is not limited to, the following steps 1510-1530:

[0222] Step 1510: Determine the virtual digest node corresponding to each local cache node;

[0223] Step 1520: For each virtual summary node, calculate the difference between the block index and the node summary information to obtain the summary difference;

[0224] Step 1530: Based on the digest difference, determine the local cache node corresponding to the block index and store the blocks in the local cache node corresponding to the block index.

[0225] Steps 1510-1530 are described in detail below.

[0226] In step 1510, the virtual digest node corresponding to each local cache node is determined.

[0227] Each virtual summary node has node summary information, and each virtual summary node is located on the same preset summary data body.

[0228] A virtual digest node refers to a virtual node that a local cache node maps to on the digest data body.

[0229] The node summary information is used to indicate the summary value corresponding to the local cache node in the summary data body.

[0230] In this specific implementation, firstly, based on the total number of cache nodes in the local cache cluster, a slot position containing a total number of digest values ​​from the cache nodes is randomly selected on the digest data body. Next, a virtual digest node is set at each selected slot position, resulting in a total number of virtual digest nodes. Further, the local cache nodes and virtual digest nodes are randomly mapped, so that each local cache node is mapped to a virtual digest node, resulting in virtual digest nodes corresponding to each local cache node. The digest value corresponding to the slot position of each virtual digest node is then used as the node digest information of that virtual digest node (local cache node).

[0231] In step 1520, for each virtual summary node, the difference between the block index and the node summary information is calculated to obtain the summary difference.

[0232] The summary difference is used to indicate the degree of difference between the summary information of the block index and the node summary information of the virtual summary node.

[0233] In the specific implementation of this embodiment, for each virtual digest node, the difference between the block index (i.e., the digest value of the block) and the node digest information of the virtual digest node (i.e., the digest value corresponding to the slot position where the virtual digest node is located) is calculated to obtain the digest difference.

[0234] In addition, the mapping position of the block can be located on the preset summary data body according to the block index, and the summary difference can be calculated based on the difference between the mapping position and the position of each virtual summary node on the preset summary data body.

[0235] In step 1530, based on the digest difference, the local cache node corresponding to the block index is determined, and the block is stored in the local cache node corresponding to the block index.

[0236] In this specific implementation, since the smaller the digest difference between the block index and the node digest information of the virtual digest node, the closer the mapping position of the block on the digest data body is to the position of the virtual digest node, the following steps are taken: First, based on the digest difference between the block index and the node digest information of each virtual digest node, the virtual digest node corresponding to the smallest digest difference is determined, and the local cache node corresponding to this virtual digest node is determined as the local cache node corresponding to the block index. Next, the block is stored in the local cache node corresponding to the block index.

[0237] Figure 16 illustrates a specific example of storing blocks in local cache nodes corresponding to their block indices. Specifically, within a predefined digest data body, there are virtual digest nodes corresponding to local cache node 4, local cache node 1, local cache node 2, and local cache node 3. Based on this, since the block indices of blocks 5 and 3 are closer to the virtual digest node corresponding to local cache node 1 in a clockwise direction, blocks 5 and 3 are cached in local cache node 1; the block index of block 6 is closer to the virtual digest node corresponding to local cache node 2, so block 6 is cached in local cache node 2; the block indices of blocks 1 and 2 are closer to the virtual digest node corresponding to local cache node 3, so blocks 1 and 2 are cached in local cache node 3; and the block index of block 4 is closer to the virtual digest node corresponding to local cache node 4, so block 4 is cached in local cache node 4.

[0238] The advantage of this embodiment is that, based on the number of local cache nodes in the local cache cluster, multiple virtual digest nodes are constructed on the digest data body, and a mapping relationship is established between each local cache node and the virtual digest nodes. Furthermore, using digest differences, the mapping position of the block index on the digest data body and the positional relationship of each virtual digest node are quantified, and blocks are preferentially cached to the local cache nodes corresponding to the virtual digest nodes closer to their block indexes. This enables distributed caching of each block, ensuring that all data requiring caching is cached evenly. This helps reduce situations where some local cache nodes are overloaded due to excessive cached data, or underloaded due to insufficient cached data, thus improving the overall cache space utilization efficiency of the local cache cluster. Additionally, since local cache nodes and each block (cache data) are mapped to the same virtual digest data body, specific locations can be determined based on the digest value of the cached data, which is beneficial for distributed caching of cached data.

[0239] The following is a detailed description of the data processing procedure when a node change occurs in a local cache cluster according to an embodiment of this disclosure.

[0240] Referring to Figure 17, in one embodiment, the data processing procedure when a node exits in the local cache cluster may include, but is not limited to, the following steps 1710-1730:

[0241] Step 1710: If a local cache node is detected to have exited the local cache cluster, the virtual digest node corresponding to the local cache node is cleared from the digest data body, and the digest value of each block stored in the local cache node is determined.

[0242] Step 1720: Based on the summary value, locate the blocks on the summary data body to obtain the data positions of the blocks;

[0243] Step 1730: Based on the data location, restore the blocks to the local cache node corresponding to the virtual digest node whose distance from the data location meets the first condition.

[0244] Steps 1710-1730 are described in detail below.

[0245] In step 1710, if a local cache node is detected to have exited the local cache cluster, the virtual digest node corresponding to the local cache node is cleared from the digest data body, and the digest value of each block stored in the local cache node is determined.

[0246] In the specific implementation of this embodiment, if a local cache node is detected to have exited the local cache cluster, it indicates that the local cache node is no longer suitable for caching any data. Therefore, the virtual digest node corresponding to the local cache node is first cleared from the digest data body to ensure that the virtual digest node on the digest data body matches the actual situation of the local cache cluster. Next, for each block stored in the local cache node, the digest value of each block is determined.

[0247] In step 1720, based on the digest value, blocks are located on the digest data body to obtain the data location of the blocks.

[0248] Data location is used to indicate the specific location where the chunk should be placed on the summary data body.

[0249] In the specific implementation of this embodiment, the blocks are mapped to the summary data body according to the summary value to achieve the positioning of the blocks; since the summary data body is composed of multiple slots containing summary values, the slot position corresponding to the summary value of the block is used as the data position of the block on the summary data body.

[0250] In step 1730, based on the data location, the blocks are re-stored on the local cache node corresponding to the virtual digest node whose distance from the data location meets the first condition.

[0251] Distance is used to indicate the degree of difference in position between the digest value of the chunk and the virtual digest node corresponding to the local cache node that exited the local cache cluster on the preset digest data body.

[0252] The first condition is used to limit the position requirements of the chunk and the local cache node to be cached on the preset digest data body.

[0253] In this specific implementation, firstly, the distance from the data location of the block to each virtual digest node is calculated in a clockwise direction to obtain the block distance. Then, among the multiple virtual digest nodes, the local cache node corresponding to the virtual digest node with the smallest block distance (the virtual digest node closest to the block's block index) is selected to cache the block.

[0254] The advantage of this embodiment is that it takes into account the scenario where cached data needs to be migrated when a local cache node leaves the local cache cluster. For each block stored in the local cache node, based on the clockwise distance between the block index and each virtual digest node on the digest data body, the block is preferentially cached on the nearest local cache node, which can realize the rapid migration of cached data and improve the cache security and stability of cached data.

[0255] Referring to Figure 18, in one embodiment, the data processing procedure when a new node is added to the local cache cluster may include, but is not limited to, the following steps 1810-1840:

[0256] Step 1810: If a new cache node is detected in the local cache cluster, determine the new summary node corresponding to the new cache node in the summary data body;

[0257] Step 1820: Determine the node distance between the newly added summary node and each virtual summary node;

[0258] Step 1830: Based on node distance, determine the target digest node among all virtual digest nodes, and determine the local cache node corresponding to the target digest node as the base cache node;

[0259] Step 1840: The multiple blocks stored in the baseline cache node are redistributed between the new cache node and the baseline cache node.

[0260] Steps 1810-1840 are described in detail below.

[0261] In step 1810, if a new cache node is detected in the local cache cluster, the new summary node corresponding to the new cache node is determined in the summary data body.

[0262] In the specific implementation of this embodiment, if a new cache node is detected in the local cache cluster, a new cache node is generated by randomly selecting a slot position other than the existing virtual cache node on the digest data body, and the mapping relationship between the new cache node and the new cache node is determined.

[0263] In step 1820, the node distance between the newly added summary node and each virtual summary node is determined.

[0264] Node distance is used to indicate the degree of difference in position between a newly added summary node and a previously existing virtual summary node on a preset summary data body.

[0265] In this specific implementation, the location of the newly added summary node is located on the preset summary data body, and the node distance between the newly added summary node and each virtual summary node is calculated based on the difference between the location of the newly added summary node and the location of each virtual summary node on the preset summary data body. The node distance can be represented by the number of slots the newly added summary node must traverse clockwise to reach each virtual summary node.

[0266] In step 1830, based on the node distance, the target digest node is determined among the virtual digest nodes, and the local cache node corresponding to the target digest node is determined as the base cache node.

[0267] In the specific implementation of this embodiment, firstly, based on node distance, the virtual digest node with the smallest node distance is determined as the target digest node. Next, according to the correspondence between each local cache node and the virtual digest node, the local cache node corresponding to the target digest node is determined as the base cache node.

[0268] In step 1840, the multiple blocks stored in the baseline cache node are redistributed between the new cache node and the baseline cache node.

[0269] In a specific implementation of this embodiment, step 1840 may include, but is not limited to, the following steps:

[0270] Determine the first cache size of the newly added cache node, the second cache size of the baseline cache node, and the number of blocks stored in the baseline cache node;

[0271] The first ratio is determined based on the first cache size and the second cache size;

[0272] Based on the first ratio and the number of blocks, the blocks are redistributed between the new cache nodes and the baseline cache nodes.

[0273] The first cache size indicates the total amount of data that a newly added cache node is allowed to cache, and the second cache size indicates the total amount of data that the baseline cache node is allowed to cache. The number of blocks indicates the total number of blocks already cached by the baseline cache node. The first ratio is determined based on the ratio of the first cache size to the second cache size.

[0274] Specifically, firstly, the amount of cached data that each of the new cache node and the base cache node can accommodate is statistically analyzed to obtain the first cache size of the new cache node and the second cache size of the base cache node. The number of blocks cached by the base cache node is then counted to obtain the number of blocks. Next, the first cache size is divided by the second cache size to obtain a first ratio. Further, based on the first ratio and the number of blocks, the blocks are divided to obtain a first sub-number of blocks to be cached on the new cache node and a second sub-number of blocks to be cached on the base cache node. The sum of the first and second sub-numbers is the total number of blocks. Finally, the first sub-number of blocks are cached on the new cache node, and the second sub-number of blocks are cached on the base cache node, wherein the sum of the data sizes of the first sub-number of blocks does not exceed the first cache size.

[0275] The advantage of this embodiment is that it takes into account the situation where a portion of cached data needs to be migrated when a new local cache node is added to the local cache cluster. For the local cache node that is closest to the new cache node in terms of the digest data body, a portion of the cached data of the local cache node (base cache node) is migrated to the new cache node. This can effectively alleviate the cache pressure on the original local cache node when a node is added, and improve the load balance of each local cache node in the local cache cluster.

[0276] Referring to Figure 19, in one embodiment, the data processing procedure when a node failure occurs in the local cache cluster may include, but is not limited to, the following steps 1910-1930:

[0277] Step 1910: If a faulty cache node is detected in the local cache cluster, locate the virtual digest node corresponding to the faulty cache node on the digest data body to obtain the target node location of the virtual digest node corresponding to the faulty cache node.

[0278] Step 1920: Based on the target node location and the node location of the virtual digest node corresponding to each local cache node, determine multiple first cache nodes among multiple local cache nodes;

[0279] Step 1930: The blocks stored in the faulty cache node are redistributed among multiple first cache nodes.

[0280] Steps 1910-1930 are described in detail below.

[0281] In step 1910, if a faulty cache node is detected in the local cache cluster, the virtual digest node corresponding to the faulty cache node is located on the digest data body to obtain the target node location of the virtual digest node corresponding to the faulty cache node.

[0282] A faulty cache node refers to a local cache node in a local cache cluster that is unable to function properly.

[0283] The target node location is used to indicate the specific location of the virtual digest node corresponding to the fault cache node in the digest data body.

[0284] In the specific implementation of this embodiment, if a faulty cache node is detected in the local cache cluster, the virtual digest node corresponding to the faulty cache node is located on the digest data body according to the mapping relationship between each local cache node and the virtual digest node, and the slot position where the virtual digest node is located is determined as the target node position of the virtual digest node corresponding to the faulty cache node.

[0285] In step 1920, based on the target node location and the node location of the virtual digest node corresponding to each local cache node, multiple first cache nodes are determined among the multiple local cache nodes.

[0286] In the specific implementation of this embodiment, based on the target node position and the node position of the virtual digest node corresponding to each local cache node, the local cache node corresponding to the virtual digest node at the second number of node positions closest to the target node position in the clockwise direction is taken as the first cache node. The second number can be set according to actual business needs and is not limited.

[0287] In step 1930, the blocks stored in the faulty cache node are redistributed among multiple first cache nodes.

[0288] In a specific implementation of this embodiment, step 1930 may include, but is not limited to, the following steps:

[0289] Determine the cache node status of each first cache node and the block information of the blocks stored in the faulty cache node;

[0290] Based on the cache node status and block information, a second cache node is selected from multiple first cache nodes, and the blocks stored in the faulty cache node are re-stored in the second cache node.

[0291] The cache node status indicates the running status of the first cache node, including node latency and the amount of data allowed to be cached. The block information indicates the size of the data blocks that need to be re-cached.

[0292] Specifically, firstly, based on a comparison between the allowed cached data size in the cache node status and the data size in the block information, the first cache node with an allowed cached data size not less than the sum of the data sizes of all blocks to be re-cached is selected as an intermediate cache node. Next, among the multiple intermediate cache nodes, the intermediate cache node with the lowest node latency is selected as the second cache node, and the blocks stored in the failed cache node are re-stored in the second cache node.

[0293] Furthermore, in order to ensure that the local cache cluster maintains load balancing after all blocks stored in the failed cache node are re-cached, the specific process of redistributing the blocks in the failed cache node among multiple first cache nodes can be similar to the specific process of redistributing blocks in step 1840. For the sake of brevity, it will not be elaborated further.

[0294] The advantage of this embodiment is that it considers the scenario where cached data needs to be redistributed when a node fails in the local cache cluster. Based on the location of the virtual digest node corresponding to the failed cache node, it prioritizes selecting multiple local cache nodes closer to the failed cache node as first cache nodes. Then, based on the node operating status of each first cache node and the block information of each block to be re-cached, it re-caches each block, effectively reducing the risk of cached data loss and improving the integrity and non-volatility of the cached data in the local cache cluster. The decentralized local cache cluster based on consistent digests in this embodiment effectively avoids single points of failure, improves the fault tolerance of the entire cache system, and enhances the overall availability and stability of the cache system. Because the cached data is more evenly distributed in the local cache cluster, even if the number of local cache nodes changes, the distribution of cached data can still remain relatively balanced, thereby avoiding the problem of load concentration on a few nodes.

[0295] The following is a detailed description of the process of updating cached data in a local cache cluster according to an embodiment of this disclosure.

[0296] Since the cache space of a local cache cluster is often limited, and the usage patterns of different blocks (cache data) often differ, allowing infrequently used or rarely used blocks to be cached in the local cache cluster for extended periods can lead to inefficient cache space utilization and reduced cache cluster efficiency. Therefore, this disclosure provides a scheme for eviction of cached data based on the number of blocks on a local cache node. This scheme enables timely updates to the cached data in the local cache cluster, improving cache space utilization efficiency.

[0297] Referring to Figure 20, in one embodiment, the specific process of updating the cached data of the local cache cluster may include, but is not limited to, the following steps 2010-2020:

[0298] Step 2010: For each block stored in each local cache node, determine the timestamp of the block being invoked;

[0299] Step 2020: If the number of blocks in the local cache node reaches the second condition, then based on the called timestamp, determine the block to be cleared from the multiple blocks stored in the local cache node, and clear the block to be cleared in the local cache cluster.

[0300] The following is a detailed description of steps 2010-2020.

[0301] In step 2010, the timestamp of the block being invoked is determined for each block stored in each local cache node.

[0302] The call timestamp is used to indicate the specific time when the chunk was last called.

[0303] In this specific implementation, the cache server of the local cache cluster records the specific time point of each call to each block stored on each local cache node in the local cache cluster. Based on this, with authorization, the time point of the most recent call to each block can be retrieved from the background logs of the cache server of the local cache cluster, thus obtaining the call timestamp of the block.

[0304] In step 2020, if it is determined that the number of blocks in the local cache node reaches the second condition, then based on the called timestamp, the block to be cleared is determined from the multiple blocks stored in the local cache node, and the block to be cleared is cleared in the local cache cluster.

[0305] The second condition is used to determine whether the cached data on the local cache nodes in the local cache cluster has reached the requirement to trigger data clearing.

[0306] Blocks to be cleared refer to blocks whose timestamps do not meet the requirements and need to be eliminated.

[0307] In this specific implementation, firstly, for each local cache node in the local cache cluster, the number of cached blocks on the local cache node is counted to obtain the number of blocks on the local cache node. Next, the number of blocks is compared with the block threshold defined in the second condition. If the number of blocks is greater than or equal to the block threshold, it indicates that there are too many cached blocks on the local cache node, and the number of blocks in the local cache node has reached the second condition, requiring data clearing of the blocks on the local cache node. Based on this, based on the call timestamps of each block stored in the local cache node, a predetermined number of blocks with earlier call timestamps are selected as blocks to be cleared, and these blocks are cleared in the local cache cluster. The block threshold is used to limit the minimum number of blocks required to trigger data clearing, and the predetermined number indicates the number of blocks to be cleared in a single cache data eviction.

[0308] Furthermore, if the number of blocks is less than the block threshold, it indicates that there are fewer blocks cached on the local cache node, and the number of blocks in the local cache node has not reached the second condition, so there is no need to clear the data on the blocks on the local cache node.

[0309] Figure 22A illustrates the process of evicting blocks from local cache node K based on their call timestamps. Specifically, local cache node K caches eight blocks: block 1 (call timestamp 21:27), block 21 (call timestamp 18:46), block 3 (call timestamp 14:07), block 14 (call timestamp 13:21), block 23 (call timestamp 11:03), block 6 (call timestamp 09:24), block 8 (call timestamp 07:58), and block 17 (call timestamp 07:12). Therefore, when evicting the three blocks with the earliest call timestamps from local cache node K, blocks 6, 8, and 17 are cleared.

[0310] The advantage of this embodiment is that it takes into account that if a certain block (cache data) has been recently accessed, it is more likely to be accessed in the future. It identifies the blocks that have been accessed recently and the blocks that have not been accessed recently based on the access timestamps of each block. Therefore, when the cache data eviction mechanism is triggered, the least recently accessed blocks are cleared. This method can update the cache data of the local cache cluster in a timely manner and clear cache data that has not been accessed for a long time, thereby improving the cache space utilization efficiency of the local cache cluster.

[0311] Referring to Figure 21, in one embodiment, the specific process of updating the cached data of the local cache cluster may include, but is not limited to, the following steps 2110-2120:

[0312] Step 2110: For each block stored in each local cache node, determine the frequency of block access.

[0313] Step 2120: If the number of blocks in the local cache node reaches the second condition, then based on the frequency of being called, determine the blocks to be cleared from the multiple blocks stored in the local cache node, and clear the blocks to be cleared in the local cache cluster.

[0314] Steps 2110-2120 are described in detail below.

[0315] In step 2110, the frequency of each block being called is determined for each block stored in each local cache node.

[0316] The call frequency indicates the total number of times a chunk has been called before the current time.

[0317] In this specific implementation, the cache server of the local cache cluster continuously accumulates the number of times each block is called on each local cache node in the local cache cluster. Based on this, with authorization, the total number of times each block has been called before the current time can be retrieved from the background log of the cache server of the local cache cluster, thus obtaining the call frequency of the block.

[0318] In step 2120, if it is determined that the number of blocks in the local cache node reaches the second condition, then based on the frequency of being called, the blocks to be cleared are determined from the multiple blocks stored in the local cache node, and the blocks to be cleared are cleared in the local cache cluster.

[0319] In this specific implementation, firstly, for each local cache node in the local cache cluster, the number of cached blocks on the local cache node is counted to obtain the number of blocks on the local cache node. Next, the number of blocks is compared with the block threshold defined in the second condition. If the number of blocks is greater than or equal to the block threshold, it indicates that there are too many cached blocks on the local cache node, and the number of blocks in the local cache node has reached the second condition, requiring data clearing of the blocks on the local cache node. Based on this, according to the frequency of calls to each block stored in the local cache node, a predetermined number of blocks with lower call frequencies are selected as blocks to be cleared, and these blocks are cleared in the local cache cluster. The block threshold is used to limit the minimum number of blocks required to trigger data clearing, and the predetermined number indicates the number of blocks to be cleared in a single cache data eviction.

[0320] Figure 22B illustrates the process of evicting blocks from local cache node K based on their call frequency. Specifically, local cache node K caches eight blocks: block 1 (call frequency 131), block 21 (call frequency 13), block 3 (call frequency 22), block 14 (call frequency 28), block 23 (call frequency 80), block 6 (call frequency 74), block 8 (call frequency 18), and block 17 (call frequency 43). Therefore, when evicting the three least frequently called blocks from local cache node K, blocks 21, 3, and 8 are cleared.

[0321] The advantage of this embodiment is that it takes into account that if a certain block (cache data) was frequently accessed before the current time, then that block will likely be accessed frequently in the future. Therefore, based on the access frequency of each block, it distinguishes between blocks that were frequently accessed before the current time and blocks that were accessed less frequently before the current time. Thus, when the cache data eviction mechanism is triggered, the least accessed blocks are cleared. This approach can update the cache data of the local cache cluster in a timely manner and clear the less accessed cache data, thereby improving the cache space utilization efficiency of the local cache cluster.

[0322] In another embodiment, the specific process of updating the cached data of the local cache cluster may include, but is not limited to, the following steps:

[0323] For each block stored in each local cache node, determine the expiration time of each block;

[0324] If it is determined that the current time has expired, the block is cleared in the local cache cluster.

[0325] The expiration time refers to the specific point in time when a block should be removed from the local cache cluster.

[0326] In this specific implementation, the expiration time of each block is pre-set before being cached on the local cache node and is bound to each block. Based on this, with authorization, the block information of each block can be obtained from the background logs of the cache server in the local cache cluster, and the expiration time of the block can be extracted from the block information. The expiration time indicates the effective period of the block. Next, for each block, the current time is checked in real-time to see if it has reached its respective expiration time. If it is determined that the current time has reached the expiration time, indicating that the block's effective period has expired and it is no longer allowed to be accessed, the block is cleared from the local cache cluster. If the current time has not reached the expiration time, the block continues to be cached on the local cache node, so that the block can still be accessed at any time.

[0327] The advantage of this embodiment is that when each block is formed, an expiration time is set for each block, and the difference between the current time and the expiration time is detected in real time. When the current time reaches the expiration time, the block is cleared in the local cache cluster. This method can promptly clear expired blocks on the local cache nodes, thereby improving the rationality of block management and the space utilization efficiency of the local cache cluster.

[0328] The apparatus and device according to embodiments of this disclosure will now be described.

[0329] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0330] It should be noted that in various specific embodiments of this application, when processing is required based on data related to the characteristics of the target object, such as target object attribute information or a set of attribute information, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require obtaining target object attribute information, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of the embodiments of this application be obtained.

[0331] Figure 23 is a schematic diagram of the structure of the cache data processing device 2300 provided in an embodiment of this disclosure. The cache data processing device 2300 includes:

[0332] The first acquisition unit 2310 is used to acquire the processing request of the processing node for the target block in the target document, wherein the target document is pre-divided into multiple blocks, the multiple blocks include the target block, and the processing request contains the block index of the target block.

[0333] The determining unit 2320 is used to determine the target local cache node corresponding to the block index in the local cache cluster bound to the processing node, wherein the local cache cluster contains multiple local cache nodes, and the multiple local cache nodes contain the target local cache node.

[0334] The second acquisition unit 2330 is used to acquire the target block from the target local cache node if the target local cache node stores the target block;

[0335] The third acquisition unit 2340 is used to acquire the target block from the data source if the target local cache node does not store the target block.

[0336] Optionally, the cache data processing apparatus 2300 further includes a cache unit (not shown), which includes:

[0337] A partitioning module (not shown) is used to divide pages in the target document into multiple blocks along the dimensions of rows and columns;

[0338] A determination module (not shown) is used to determine the chunk index of a chunk based on the chunk's digest value;

[0339] A storage module (not shown) is used to store blocks into the local cache node corresponding to the block index.

[0340] Optionally, the determining module (not shown) is used for:

[0341] The remainder of the digest value modulo a first number is used as the block index, where the first number is the number of local cache nodes in the local cache cluster;

[0342] The storage module (not shown) is used for:

[0343] The blocks are stored in local cache nodes with the same sequence number as the block index.

[0344] Optionally, the determining module (not shown) is used for:

[0345] The summary value is used as the block index;

[0346] Local cache nodes have block index ranges, and the storage module (not shown) is used for:

[0347] The blocks are stored in the local cache node corresponding to the block index range to which the block index belongs.

[0348] Optionally, the block index range of the local cache node is determined in the following way:

[0349] Obtain the preset summary data body;

[0350] Determine the total number of cache nodes in the local cache cluster;

[0351] Based on the total number of cache nodes, the digest data body is divided into multiple digest data segments.

[0352] Multiple local cache nodes are matched with multiple digest data segments so that each local cache node corresponds to a digest data segment, and the digest data segment is used as the block index range corresponding to the local cache node.

[0353] Optionally, the determining module (not shown) is used for:

[0354] The summary value is used as the block index;

[0355] The storage module (not shown) is used for:

[0356] Determine the virtual digest node corresponding to each local cache node. Each virtual digest node has node digest information and is located on the same preset digest data body.

[0357] For each virtual summary node, the difference between the block index and the node summary information is calculated to obtain the summary difference.

[0358] Based on the digest difference, the local cache node corresponding to the block index is determined, and the block is stored in the local cache node corresponding to the block index.

[0359] Optionally, the cached data processing apparatus 2300 further includes an exit unit (not shown), which is used for:

[0360] If a local cache node is detected to have exited the local cache cluster, the virtual digest node corresponding to the local cache node is cleared from the digest data body, and the digest value of each block stored in the local cache node is determined.

[0361] Based on the summary value, the blocks are located on the summary data body to obtain the data position of the blocks;

[0362] Based on the data location, the blocks are re-stored on the local cache node corresponding to the virtual digest node whose distance from the data location meets the first condition.

[0363] Optionally, the cache data processing apparatus 2300 further includes a new unit (not shown), which is used for:

[0364] If a new cache node is detected in the local cache cluster, the new summary node corresponding to the new cache node is determined in the summary data body;

[0365] Determine the node distance between the newly added summary node and each virtual summary node;

[0366] Based on node distance, the target digest node is determined among all virtual digest nodes, and the local cache node corresponding to the target digest node is determined as the baseline cache node;

[0367] The multiple blocks stored in the baseline cache node are redistributed between the new cache node and the baseline cache node.

[0368] Optionally, multiple blocks in the baseline cache node are reallocated between the new cache node and the baseline cache node, including:

[0369] Determine the first cache size of the newly added cache node, the second cache size of the baseline cache node, and the number of blocks in the partition;

[0370] The first ratio is determined based on the first cache size and the second cache size;

[0371] Based on the first ratio and the number of blocks, the blocks are redistributed between the new cache nodes and the baseline cache nodes.

[0372] Optionally, the cached data processing apparatus 2300 further includes a fault unit (not shown), which is used for:

[0373] If a faulty cache node is detected in the local cache cluster, the virtual digest node corresponding to the faulty cache node is located on the digest data body to obtain the target node location of the virtual digest node corresponding to the faulty cache node.

[0374] Based on the target node location and the node location of the virtual digest node corresponding to each local cache node, multiple first cache nodes are determined in each local cache node;

[0375] The blocks stored in the faulty cache node are redistributed among multiple first cache nodes.

[0376] Optionally, the blocks stored in the faulty cache node are redistributed among multiple first cache nodes, including:

[0377] Determine the cache node status of each first cache node and the block information of the blocks stored in the faulty cache node;

[0378] Based on the cache node status and block information, a second cache node is selected from multiple first cache nodes, and the blocks stored in the faulty cache node are re-stored in the second cache node.

[0379] Optionally, the cached data processing device 2300 further includes a first update unit (not shown), which is used for:

[0380] For each block stored in each local cache node, determine the timestamp when the block was invoked;

[0381] If the number of blocks in the local cache node reaches the second condition, then based on the timestamp of the call, the block to be cleared is determined from the multiple blocks stored in the local cache node, and the block to be cleared is cleared in the local cache cluster.

[0382] Optionally, the cached data processing apparatus 2300 further includes a second update unit (not shown), which is used for:

[0383] For each block stored in each local cache node, determine the frequency of each block being accessed;

[0384] If the number of blocks in the local cache node reaches the second condition, then based on the frequency of being called, the block to be cleared is determined from the multiple blocks stored in the local cache node, and the block to be cleared is cleared in the local cache cluster.

[0385] Optionally, the cached data processing apparatus 2300 further includes a third update unit (not shown), which is used for:

[0386] For each block stored in each local cache node, determine the expiration time of each block;

[0387] If it is determined that the current time has expired, the block is cleared in the local cache cluster.

[0388] Optionally, the cached data processing apparatus 2300 further includes a pre-storage unit (not shown), which is used for:

[0389] Determine the data retrieval frequency and data size for each of the multiple reference data sets;

[0390] Based on the frequency of data access and the size of the data, alternative data is determined from multiple reference data, and the alternative data is cached in the local cache cluster.

[0391] Optionally, the processing request for the target chunk is generated by the processing node based on the received processing task for the target document. The processing task for the target document is allocated to the processing node by the task scheduling node in the task cluster through the following process:

[0392] Get the current number of tasks and the current node status of each candidate processing node in the task cluster;

[0393] For each candidate processing node, a first score is determined based on the current number of tasks;

[0394] Determine the second score based on the current node state;

[0395] Based on the first score and the second score, determine the total node score of the candidate processing node;

[0396] Based on the total node score, the processing node for performing the processing task for the target document is determined from multiple candidate processing nodes, and the processing task is assigned to that processing node.

[0397] Optionally, the current node status of the candidate processing node includes the node latency and node resource status.

[0398] Based on the current node state, the second score is determined, including:

[0399] The first sub-score is determined based on the node latency.

[0400] The second sub-score is determined based on the node resource status;

[0401] The second score of the candidate processing node is determined based on the weighted sum of the first and second sub-scores.

[0402] Referring to Figure 24, which is a structural block diagram of a portion of a terminal implementing the cached data processing method of this disclosure, the terminal includes: a radio frequency (RF) circuit 2410, a memory 2415, an input unit 2430, a display unit 2440, a sensor 2450, an audio circuit 2460, a wireless fidelity (WiFi) module 2470, a processor 2480, and a power supply 2490, among other components. Those skilled in the art will understand that the terminal structure shown in Figure 24 does not constitute a limitation on a mobile phone or computer, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0403] The RF circuit 2410 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 2480; in addition, it transmits uplink data to the base station.

[0404] The memory 2415 can be used to store software programs and modules. The processor 2480 executes various functional applications and data processing of the target terminal by running the software programs and modules stored in the memory 2415.

[0405] The input unit 2430 can be used to receive input numeric or character information, and to generate key signal inputs related to the settings and function control of the target terminal. Specifically, the input unit 2430 may include a touch panel 2431 and other input devices 2432.

[0406] Display unit 2440 can be used to display input or provided information, as well as various menus of the target terminal. Display unit 2440 may include display panel 2441.

[0407] Audio circuitry 2460, speaker 2461, and microphone 2462 provide an audio interface.

[0408] In this embodiment, the processor 2480 included in the terminal can execute the cached data processing method of the previous embodiment.

[0409] The terminals disclosed in this embodiment include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The embodiments of this invention can be applied to various scenarios, including but not limited to data security, blockchain, data storage, and information technology.

[0410] Figure 25 is a partial structural block diagram of a server implementing the cached data processing method of this disclosure. The server can vary significantly due to different configurations or performance, and may include one or more Central Processing Units (CPUs) 2522 (e.g., one or more processors) and a memory 2532, and one or more storage media 2530 (e.g., one or more mass storage devices) for storing application programs 2542 or data 2544. The memory 2532 and storage media 2530 can be temporary or persistent storage. The program stored in the storage media 2530 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the server. Furthermore, the CPU 2522 may be configured to communicate with the storage media 2530 and execute the series of instruction operations in the storage media 2530 on the server.

[0411] The server may also include one or more power supplies 2523, one or more wired or wireless network interfaces 2550, one or more input / output interfaces 2558, and / or one or more operating systems 2541, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0412] The central processing unit 2522 in the server can be used to execute the cached data processing method of the present disclosure embodiments.

[0413] This disclosure also provides a computer-readable storage medium for storing program code for executing the cached data processing methods of the foregoing embodiments.

[0414] This disclosure also provides a computer program product comprising a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the cached data processing method described above.

[0415] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0416] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0417] It should be understood that in the description of the embodiments of this disclosure, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0418] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0419] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0420] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0421] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0422] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0423] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0424] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A method for processing cached data, executed by an electronic device, the method comprising: The processing node obtains a processing request for a target block in the target document, wherein the target document is pre-divided into multiple blocks, the multiple blocks include the target block, and the processing request contains the block index of the target block; In the local cache cluster bound to the processing node, the target local cache node corresponding to the block index is determined, wherein the local cache cluster contains multiple local cache nodes, and the multiple local cache nodes include the target local cache node; If the target local cache node stores the target block, retrieve the target block from the target local cache node; If the target local cache node does not store the target block, the target block is obtained from the data source.

2. The cached data processing method according to claim 1, wherein, Before the processing node obtains the processing request for the target chunk in the target document, the method further includes: The pages in the target document are divided into the multiple blocks in terms of rows and columns; Based on the digest value of the block, determine the block index; The blocks are stored in the local cache node corresponding to the block index.

3. The cached data processing method according to claim 2, wherein, Determining the block index based on the digest value of the block includes: The remainder of the digest value modulo a first number is determined and used as the block index, wherein the first number is the number of local cache nodes in the local cache cluster; The step of storing the block in the local cache node corresponding to the block index includes: The block is stored in the local cache node whose sequence number matches the block index.

4. The cached data processing method according to claim 2, wherein, Determining the block index based on the digest value of the block includes: The summary value is determined as the block index; The local cache node has a block index range, and storing the block in the local cache node corresponding to the block index includes: The blocks are stored in the local cache node corresponding to the block index range to which the block index belongs.

5. The cached data processing method according to claim 4, wherein, The block index range of the local cache node is determined in the following way: Obtain the preset summary data body; Determine the total number of cache nodes in the local cache cluster; Based on the total number of cache nodes, the digest data body is divided into multiple digest data segments. The plurality of local cache nodes are matched with the plurality of digest data segments so that each local cache node corresponds to one digest data segment, and the digest data segment is used as the block index range corresponding to the local cache node.

6. The cached data processing method according to claim 2, wherein, Determining the block index based on the digest value of the block includes: The summary value is determined as the block index; The step of storing the block in the local cache node corresponding to the block index includes: Each of the local cache nodes is determined to have its own corresponding virtual digest node, wherein each virtual digest node has node digest information and each virtual digest node is located on the same preset digest data body; For each virtual summary node, the difference between the block index and the node summary information is calculated to obtain the summary difference. Based on the digest difference, the local cache node corresponding to the block index is determined, and the block is stored in the local cache node corresponding to the block index.

7. The cached data processing method according to claim 6, wherein, After storing the block in the local cache node corresponding to the block index, the method further includes: If it is detected that the local cache node has exited the local cache cluster, the virtual digest node corresponding to the local cache node is cleared from the digest data body, and the digest value of each block stored in the local cache node is determined. Based on the summary value, the block is located on the summary data body to obtain the data position of the block; Based on the data location, the block is re-stored on the local cache node corresponding to the virtual digest node whose distance from the data location meets the first condition.

8. The cached data processing method according to claim 6 or 7, wherein, After storing the block in the local cache node corresponding to the block index, the method further includes: If a new cache node is detected in the local cache cluster, the new summary node corresponding to the new cache node is determined in the summary data body; Determine the node distance between the newly added summary node and each of the virtual summary nodes; Based on the node distance, a target digest node is determined among the virtual digest nodes, and the local cache node corresponding to the target digest node is determined as the base cache node; The multiple blocks stored in the baseline cache node are redistributed between the new cache node and the baseline cache node.

9. The cached data processing method according to any one of claims 6 to 8, wherein, After storing the block in the local cache node corresponding to the block index, the method further includes: If a faulty cache node is detected in the local cache cluster, the virtual digest node corresponding to the faulty cache node is located on the digest data body to obtain the target node location of the virtual digest node corresponding to the faulty cache node. Based on the target node location and the node location of the virtual digest node corresponding to each of the local cache nodes, a plurality of first cache nodes are determined in each of the local cache nodes; The blocks stored in the fault cache node are redistributed among the plurality of first cache nodes.

10. The cached data processing method according to claim 9, wherein, The step of redistributing the blocks stored in the faulty cache node among the plurality of first cache nodes includes: Determine the cache node status of each of the first cache nodes and the block information of the blocks stored in the faulty cache node; Based on the cache node status and the block information, a second cache node is selected from the plurality of first cache nodes, and the blocks stored in the faulty cache node are re-stored in the second cache node.

11. The cached data processing method according to any one of claims 1 to 10, wherein, The blocks stored in the plurality of local cache nodes in the local cache cluster are updated in the following manner: For each block stored in each of the local cache nodes, determine the timestamp of the block being invoked; If it is determined that the number of blocks in the local cache node reaches the second condition, then based on the invoked timestamp, the block to be cleared is determined from the multiple blocks stored in the local cache node, and the block to be cleared is cleared in the local cache cluster.

12. The cached data processing method according to any one of claims 1 to 10, wherein, The blocks stored in the plurality of local cache nodes in the local cache cluster are updated in the following manner: For each block stored in each of the local cache nodes, determine the frequency of invocation of the block; If it is determined that the number of blocks in the local cache node reaches the second condition, then based on the frequency of being called, a block to be cleared is determined from the multiple blocks stored in the local cache node, and the block to be cleared is cleared in the local cache cluster.

13. The cached data processing method according to any one of claims 1 to 10, wherein, The blocks stored in the plurality of local cache nodes in the local cache cluster are updated in the following manner: For each block stored in each of the local cache nodes, determine the expiration time of each block; If it is determined that the current time has reached the expiration time, then the block is cleared in the local cache cluster.

14. The cached data processing method according to any one of claims 1 to 13, wherein, Before the processing node obtains the processing request for the target chunk in the target document, the method further includes: Determine the data retrieval frequency and data size for each of the multiple reference data sets; Based on the data call frequency and the data size, candidate data is determined from the plurality of reference data, and the candidate data is cached in the local cache cluster.

15. A cached data processing apparatus, wherein, The device includes: The first acquisition unit is used to acquire the processing request of the processing node for the target block in the target document, wherein the target document is pre-divided into multiple blocks, the multiple blocks include the target block, and the processing request contains the block index of the target block; The determining unit is configured to determine the target local cache node corresponding to the block index in a local cache cluster bound to the processing node, wherein the local cache cluster includes multiple local cache nodes, and the multiple local cache nodes include the target local cache node; The second acquisition unit is configured to acquire the target block from the target local cache node if the target local cache node stores the target block. The third acquisition unit is used to acquire the target block from the data source if the target local cache node does not store the target block.

16. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein, When the processor executes the computer program, it implements the cached data processing method according to any one of claims 1 to 14.

17. A computer-readable storage medium storing a computer program, wherein, When the computer program is executed by the processor, it implements the cached data processing method according to any one of claims 1 to 14.

18. A computer program product comprising a computer program that is read and executed by a processor of an electronic device, causing the electronic device to perform the cached data processing method according to any one of claims 1 to 14.

Images