Cache data processing method, related device and medium

Abstract

The invention provides a cache data processing method, a related device and a medium. The method comprises the steps that a processing request of a target processing node for a target block in a target document is obtained, a page in the target document is divided into multiple blocks in advance from the dimension of rows and columns, the multiple blocks comprise the target block, and the processing request comprises a block index of the target block; in a local cache cluster bound with the target processing node, a target local cache node corresponding to the block index is determined, the local cache cluster comprises a plurality of local cache nodes, and the plurality of local cache nodes comprise the target local cache node; if the target local cache node contains the target block, obtaining the target block from the target local cache node; and if the target local cache does not contain the target block, obtaining the target block from the data source. The document transmission time in document processing can be shortened, and the document transmission efficiency can be improved. The method can be applied to various scenes such as computers, artificial intelligence and cloud technologies.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus and medium for caching data processing. Background Technology

[0002] Currently, processing nodes in a network environment typically retrieve one or more documents from a data source on a per-document basis when processing documents. Since retrieving documents from the data source every time is inefficient, a local cache cluster is usually set up within the processing node or associated with it. The local cache cluster consists of multiple cache nodes. Each cache node pre-retrieves and stores one or more documents. When a processing node processes a document, it can retrieve the document from the corresponding cache node. If the corresponding cache node fails, the entire document cannot be recovered. Furthermore, when only a portion of a document's data needs to be processed, the entire document must be retrieved for processing. This results in long document transmission times and low retrieval efficiency. Summary of the Invention

[0003] This disclosure provides a cached data processing method, related apparatus, and medium that can reduce document transmission time and improve document transmission efficiency in document processing.

[0004] According to one aspect of this disclosure, a method for processing cached data is provided, the method comprising:

[0005] Obtain the processing request of the target processing node for the target block in the target document, wherein the page in the target document is pre-divided into multiple blocks in the dimensions of rows and columns, the multiple blocks include the target block, and the processing request contains the block index of the target block;

[0006] In the local cache cluster bound to the target processing node, the target local cache node corresponding to the block index is determined, wherein the local cache cluster contains multiple local cache nodes, and the multiple local cache nodes include the target local cache node;

[0007] If the target local cache node contains the target block, obtain the target block from the target local cache node;

[0008] If the target local cache does not contain the target block, obtain the target block from the data source.

[0009] According to one aspect of this disclosure, a cached data processing apparatus is provided, the apparatus comprising:

[0010] The first acquisition unit is used to acquire the processing request of the target processing node for the target block in the target document, wherein the page in the target document is pre-divided into multiple blocks in the dimensions of rows and columns, the multiple blocks include the target block, and the processing request contains the block index of the target block;

[0011] The determining unit is configured to determine the target local cache node corresponding to the block index in a local cache cluster bound to the target processing node, wherein the local cache cluster includes multiple local cache nodes, and the multiple local cache nodes include the target local cache node;

[0012] The second acquisition unit is configured to acquire the target block from the target local cache node if the target local cache node contains the target block;

[0013] The third acquisition unit is used to acquire the target block from the data source if the target local cache does not contain the target block.

[0014] Optionally, the cached data processing device further includes a cache unit, the cache unit comprising:

[0015] A segmentation module is used to divide the pages in the target document into the multiple blocks in terms of rows and columns;

[0016] A determining module is used to determine the block index of the block based on the digest value of the block;

[0017] A storage module is used to store the blocks in the local cache node corresponding to the block index.

[0018] Optionally, the determining module is used to:

[0019] The remainder of the digest value modulo a first number is determined and used as the block index, wherein the first number is the number of local cache nodes in the local cache cluster;

[0020] The storage module is used for:

[0021] The block is stored in the local cache node whose sequence number matches the block index.

[0022] Optionally, the determining module is used to:

[0023] The summary value is determined as the block index;

[0024] The local cache node has a block index range, and the storage module is used for:

[0025] The blocks are stored in the local cache node corresponding to the block index range where the block index is located.

[0026] Optionally, the block index range of the local cache node is determined in the following way:

[0027] Obtain the preset summary data body;

[0028] Determine the total number of cache nodes in the local cache cluster;

[0029] Based on the total number of cache nodes, the digest data body is divided into multiple digest data segments.

[0030] Multiple local cache nodes are matched with multiple digest data segments so that each local cache node corresponds to one digest data segment, and the digest data segment is used as the block index range corresponding to the local cache node.

[0031] Optionally, the determining module is used to:

[0032] The summary value is determined as the block index;

[0033] The storage module is used for:

[0034] Determine the virtual digest node corresponding to each of the local cache nodes, wherein each of the virtual digest nodes has node digest information and each of the virtual digest nodes is located on the same preset digest data body;

[0035] For each virtual summary node, the difference between the block index and the node summary information is calculated to obtain the summary difference.

[0036] Based on the digest difference, the local cache node corresponding to the block index is determined, and the block is stored in the local cache node corresponding to the block index.

[0037] Optionally, the cached data processing device further includes an exit unit, the exit unit being used for:

[0038] If the target local cache node is detected to have exited the local cache cluster, the virtual digest node corresponding to the target local cache node is cleared from the digest data body, and the digest value of each block in the target local cache node is determined.

[0039] Based on the summary value, the block is located on the summary data body to obtain the data position of the block;

[0040] Based on the data location, the block is re-stored on the local cache node corresponding to the virtual digest node whose distance from the data location meets the first condition.

[0041] Optionally, the cached data processing device further includes a adding unit, the adding unit being used for:

[0042] If a new cache node is detected in the local cache cluster, the new summary node corresponding to the new cache node is determined in the summary data body;

[0043] Determine the node distance between the newly added summary node and each of the virtual summary nodes;

[0044] Based on the node distance, a target digest node is determined among the multiple virtual digest nodes, and the local cache node corresponding to the target digest node is determined as the base cache node;

[0045] The multiple blocks in the baseline cache node are redistributed among the new cache node and the baseline cache node.

[0046] Optionally, the step of reallocating the multiple blocks in the baseline cache node among the newly added cache node and the baseline cache node includes:

[0047] Determine the first cache size of the newly added cache node, the second cache size of the baseline cache node, and the number of blocks in the segment;

[0048] Based on the first cache size and the second cache size, a first ratio is determined;

[0049] Based on the first ratio and the number of blocks, the blocks are redistributed among the newly added cache nodes and the baseline cache nodes.

[0050] Optionally, the cached data processing device further includes a fault unit, the fault unit being used for:

[0051] If a node failure is detected in the local cache cluster, the virtual digest node corresponding to the failed cache node is located on the digest data body to obtain the target node location of the virtual digest node corresponding to the failed cache node.

[0052] Based on the target node location and the node location of the virtual digest node corresponding to each local cache node, the first cache node is determined among the multiple local cache nodes;

[0053] The blocks in the faulty cache node are redistributed among multiple first cache nodes.

[0054] Optionally, the step of reallocating the blocks in the faulty cache node among multiple first cache nodes includes:

[0055] Determine the cache node status of each of the first cache nodes and the block information of the blocks in the faulty cache nodes;

[0056] Based on the cache node status and the block information, a second cache node is selected from multiple first cache nodes, and the blocks in the faulty cache node are re-stored in the second cache node.

[0057] Optionally, the cached data processing device further includes a first update unit, the first update unit being used for:

[0058] For each block in each local cache node, determine the timestamp of the block being invoked;

[0059] If it is determined that the number of blocks in the local cache node reaches the second condition, then based on the invoked timestamp, the block to be cleared is determined from among the multiple blocks, and the block to be cleared is cleared in the local cache cluster.

[0060] Optionally, the cached data processing device further includes a second update unit, the second update unit being used for:

[0061] For each block in each local cache node, determine the frequency of invocation of the block;

[0062] If it is determined that the number of blocks in the local cache node reaches the second condition, then based on the frequency of being called, a block to be cleared is determined from among the multiple blocks, and the block to be cleared is cleared in the local cache cluster.

[0063] Optionally, the cached data processing device further includes a third update unit, the third update unit being used for:

[0064] For each block in each local cache node, determine the expiration time of each block;

[0065] If it is determined that the current time has reached the expiration time, then the block is cleared in the local cache cluster.

[0066] Optionally, the cached data processing device further includes a pre-storage unit, the pre-storage unit being used for:

[0067] Determine the data retrieval frequency and data size for multiple reference data sets;

[0068] Based on the data call frequency and the data size, candidate data is determined from the plurality of reference data, and the candidate data is cached in the local cache cluster.

[0069] Optionally, the processing request for the target chunk is generated by the target processing node based on the received processing task for the target document, and the processing task for the target document is allocated to the target processing node by the task scheduling node in the target task cluster through the following process:

[0070] Obtain the current number of tasks and the current node status of each candidate processing node in the target task cluster;

[0071] For each candidate processing node, a first score is determined based on the current number of tasks;

[0072] Based on the current node state, determine the second score;

[0073] Based on the first score and the second score, the total node score of the candidate processing node is determined;

[0074] Based on the total score of the nodes, the target processing node is determined from among the multiple candidate processing nodes, and the processing task is assigned to the target processing node.

[0075] Optionally, the current node state of the candidate processing node includes the node latency and node resource status of the candidate processing node;

[0076] Determining the second score based on the current node state includes:

[0077] Based on the node latency, a first sub-score is determined;

[0078] Based on the node resource status, determine the second sub-score;

[0079] The second score of the candidate processing node is determined based on the weighted sum of the first sub-score and the second sub-score.

[0080] According to one aspect of this disclosure, an electronic device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the cached data processing method as described above.

[0081] According to one aspect of this disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program that, when executed by a processor, implements the cached data processing method described above.

[0082] According to one aspect of this disclosure, a computer program product is provided, the computer program product comprising a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the cached data processing method as described above.

[0083] In this embodiment, considering the problem in related technologies where a document cannot be recovered if a cache node fails after caching it, the method pre-divides the pages of the target document into multiple blocks along the row and column dimensions when caching the target document to the local cache cluster. Caching is performed on a block-by-block basis, enabling fine-grained caching of the target document. This allows different blocks to be cached on different cache nodes. Thus, if a cache node fails, only a block of data in the target document is unrecoverable, not the entire document. Only a small block of document data needs to be recovered, improving the non-volatility of document data and reducing the difficulty of document data recovery. Furthermore, after obtaining the processing request from the target processing node for the target block in the target document, the target local cache node corresponding to the block index can be determined in the local cache cluster bound to the target processing node based on the block index contained in the processing request. This allows for quick data querying of the locally deployed local cache cluster of the target processing node, improving data query efficiency. Finally, if it is determined that the target local cache node contains the target chunk, the target chunk is directly retrieved from the local cache node; otherwise, if it is determined that the target local cache does not contain the target chunk, the target chunk is retrieved from the remote data source. This method only transmits the required target chunk when calling cached data, instead of transmitting the entire target document, which can reduce document transmission time in document processing and improve document transmission efficiency.

[0084] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0085] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0086] Figure 1 This is a system architecture diagram of a cached data processing method applied according to an embodiment of the present disclosure;

[0087] Figures 2A-2D A schematic diagram is shown illustrating the application of the cached data processing method according to an embodiment of the present disclosure in a cached data retrieval scenario;

[0088] Figure 3 This is a flowchart of a cached data processing method according to an embodiment of the present disclosure;

[0089] Figure 4 This is a flowchart illustrating the assignment of processing tasks to a target processing node according to an embodiment of the present disclosure;

[0090] Figure 5 This is a flowchart illustrating the determination of a second score based on the current node state according to an embodiment of this disclosure;

[0091] Figures 6A-6B This is a schematic diagram illustrating the process of determining the total node score of a candidate processing node according to an embodiment of the present disclosure;

[0092] Figures 7A-7B This is a schematic diagram illustrating the process of calling cached data when performing a processing task according to an embodiment of the present disclosure;

[0093] Figure 8 This is a flowchart illustrating the pre-storage of frequently used data in a local cache cluster according to an embodiment of the present disclosure;

[0094] Figure 9 This is a schematic diagram illustrating the data filtering process when frequently used data is pre-stored in a local cache cluster according to an embodiment of the present disclosure;

[0095] Figure 10 This is a flowchart illustrating the process of caching target documents in chunks to a local cache cluster according to an embodiment of this disclosure;

[0096] Figures 11A-11B This is a schematic diagram illustrating the process of dividing a target document into multiple blocks according to an embodiment of the present disclosure;

[0097] Figure 12 This is a schematic diagram illustrating the implementation process of storing blocks into the local cache node corresponding to the block index according to an embodiment of the present disclosure;

[0098] Figure 13 This is a flowchart illustrating the determination of the block index range for each local cache node according to an embodiment of the present disclosure;

[0099] Figures 14A-14B This is a schematic diagram illustrating the implementation process of storing blocks into the local cache node corresponding to the block index according to an embodiment of the present disclosure;

[0100] Figure 15 This is a flowchart illustrating the storage of blocks into local cache nodes corresponding to block indexes according to an embodiment of this disclosure;

[0101] Figure 16 This is a schematic diagram illustrating the implementation process of storing blocks into the local cache node corresponding to the block index according to an embodiment of the present disclosure;

[0102] Figure 17 This is a flowchart of cached data processing when a node exits, according to an embodiment of the present disclosure;

[0103] Figure 18 This is a flowchart illustrating cached data processing when a node is added, according to an embodiment of the present disclosure.

[0104] Figure 19 This is a flowchart illustrating cached data processing in the event of a node failure, according to an embodiment of this disclosure.

[0105] Figure 20 This is a flowchart of evicting cached data according to an embodiment of the present disclosure;

[0106] Figure 21 This is a flowchart of evicting cached data according to another embodiment of the present disclosure;

[0107] Figures 22A-22B This is a schematic diagram illustrating the process of evicting cached data according to an embodiment of the present disclosure;

[0108] Figure 23 This is a block diagram of a cached data processing apparatus according to an embodiment of the present disclosure;

[0109] Figure 24 This is a terminal structure diagram of a cached data processing method according to an embodiment of the present disclosure;

[0110] Figure 25 This is a server structure diagram of a cached data processing method according to an embodiment of the present disclosure. Detailed Implementation

[0111] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0112] The system architecture and scenarios in which this disclosure is applied are described below.

[0113] Figure 1 This is a system architecture diagram of the cached data processing method according to an embodiment of the present disclosure. It includes an object terminal 140, an Internet 130, a gateway 120, a cluster computing server 150, a local cache cluster system 110, and a remote data source 160, etc.

[0114] The target terminal 140 includes various forms such as desktop computers, laptops, PDAs (personal digital assistants), mobile phones, in-vehicle terminals, home theater terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. The target terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data. The target terminal 140 includes a data processing platform, which provides the target with the ability to initiate processing tasks for various types of data.

[0115] Cluster computing server 150 refers to a computer system that can provide certain services to object terminal 140. Compared with ordinary object terminal 140, cluster computing server 150 has higher requirements in terms of stability, security, and performance. Cluster computing server 150 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Cluster computing server 150 includes various types of services, and the implementation of each service of cluster computing server 150 is often associated with some intermediate databases or storage media. Cluster computing server 150 is used to receive processing tasks for various data initiated by objects and call cluster nodes to execute the processing tasks; local caching cluster system 110 is used to obtain various data from remote data source 160 and cache the obtained data at a small granularity; remote data source 160 is used to store various data.

[0116] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from target terminal 140 to cluster computing server 150 or local caching cluster system 110 must be forwarded to the corresponding cluster computing server 150 or local caching cluster system 110 via gateway 120. Similarly, messages sent from cluster computing server 150 or local caching cluster system 110 to target terminal 140 must also be forwarded to the corresponding target terminal 140 via gateway 120.

[0117] The embodiments disclosed herein can be applied in various scenarios, such as Figures 2A-2D The examples shown include scenarios for retrieving cached data.

[0118] like Figure 2AAs shown, when an object needs to access a portion of a document's data to perform a computational task, it triggers a cached data retrieval process on the data processing platform. At this point, the data processing platform's page displays a prompt field, "Please enter the document information associated with the data processing task," and an input area for entering the document information. Based on this, the object enters "Document Name: Document K, the index of the target document data to be retrieved is xxxxx, xxx" in the input area and clicks the "Confirm" button.

[0119] like Figure 2B As shown, after clicking on the object, a prompt field appears on the page: "The current number of tasks for each task node in the task processing cluster is as follows," showing that task node 1 currently has 12 tasks, task node 2 has 4 tasks, task node 3 has 5 tasks, and task node 4 has 8 tasks. Additionally, the page displays a prompt field: "Based on this, the current data processing task is assigned to task node 2, which will then call the target document data to execute the data processing task," indicating which specific task node is performing the data processing task.

[0120] like Figure 2C As shown, once the document data of document K has been cached at a small granularity, a prompt window will be displayed on the page. The prompt window will display the message "The target document block data of document K has been found in the local cache cluster. The target document data is being retrieved from the local cache cluster. Please wait patiently...".

[0121] like Figure 2D As shown, when the document data of document K is not cached, a prompt window will be displayed on the page. The prompt window displays the message "The target document block data of document K was not found in the local cache cluster. The document data of document K is being retrieved from the remote data source. Please wait patiently...".

[0122] The embodiments of this disclosure are described in general below.

[0123] According to one embodiment of this disclosure, a method for processing cached data is provided.

[0124] This cached data processing method is generally used in business scenarios where cached data needs to be called to perform various tasks, for example... Figures 2A-2D The illustration depicts a cached data retrieval scenario. This disclosure provides a scheme for retrieving cached data based on local cache nodes within a local cache cluster, which can reduce document transmission time and improve document transmission efficiency. The cached data processing method of this disclosure can be executed by the server corresponding to the local cache cluster.

[0125] like Figure 3 As shown, a cached data processing method according to an embodiment of this disclosure may include:

[0126] Step 310: Obtain the processing requests of the target processing node for the target block in the target document;

[0127] Step 320: In the local cache cluster bound to the target processing node, determine the target local cache node corresponding to the block index;

[0128] Step 330: If the target local cache node contains the target block, obtain the target block from the target local cache node;

[0129] Step 340: If the target local cache does not contain the target chunk, obtain the target chunk from the data source.

[0130] Steps 310-340 are described in detail below.

[0131] In step 310, the processing request of the target processing node for the target block in the target document is obtained.

[0132] A target processing node refers to a cluster node belonging to a computing cluster and used to process various tasks.

[0133] A target document refers to a document file associated with the computational task that the target processing node is to process. A portion of the document data in the target document can help the computational task to execute correctly.

[0134] In this process, the pages in the target document are pre-divided into multiple blocks along the dimensions of rows and columns, and these blocks include the target block.

[0135] A processing request refers to a request initiated by a target processing node when it wants to perform a processing task on a target document (or a computational task related to the target document). This processing request is used to request some cached data from the local cache cluster so that the processing task can be executed normally.

[0136] The processing request includes the block index of the target block.

[0137] The block index is used to identify the target block, and different blocks have different block indexes.

[0138] In step 320, the target local cache node corresponding to the block index is determined in the local cache cluster bound to the target processing node.

[0139] A local cache cluster refers to a collection of cache services deployed locally, close to the target processing node. A local cache cluster consists of multiple local cache nodes that work together to provide efficient data storage and retrieval services. The local cache cluster setup in this embodiment effectively improves data access speed and reduces the pressure on the backend database.

[0140] Among them, multiple local cache nodes contain the target local cache node.

[0141] A local cache node refers to a cluster node in a local cache cluster that has a certain amount of cache space and is capable of storing various types of data.

[0142] The target local cache node refers to the local cache node where the target block is cached.

[0143] In this specific implementation, since there is often an inherent mapping relationship between the cached data on the local cache nodes and the local cache nodes themselves in the local cache cluster, after obtaining the block index of the target block, the local cache node corresponding to the block index can be found as the target local cache node in the local cache cluster bound to the target processing node, based on the inherent mapping relationship between the cached data on the local cache nodes and the local cache nodes.

[0144] For example, a mapping relationship can be established between the block index of the cached blocks in each local cache node and the node number of the local cache node. Based on the block index, the local cache node corresponding to the node number and the block index can be used as the target local cache node.

[0145] For example, you can set the index range of each local cache node and use the local cache node with the block index located in the index range as the target local cache node.

[0146] In step 330, if the target local cache node contains the target block, the target block is obtained from the target local cache node.

[0147] In the specific implementation of this embodiment, if the target local cache node contains the target block, it indicates that the target block has been cached on the target local cache node. Therefore, the target block can be obtained directly from the target local cache node.

[0148] It should be noted that when a target block is invoked, certain specific columns can be accessed, which is beneficial for efficient reading of cached data.

[0149] In step 340, if the target local cache does not contain the target chunk, the target chunk is obtained from the data source.

[0150] Data sources refer to remote servers or cloud services that store various types of data.

[0151] In this specific implementation, if the target local cache does not contain the target chunk, it indicates that the target chunk of the target document has not been cached in the local cache cluster. Based on this, with authorization, the entire document data of the target document is obtained from the data source, and the target chunk is determined within the entire document data. Then, after caching the target document in multiple chunks in the local cache cluster, the target chunks are fed back to the target processing node.

[0152] Through steps 310-340 above, this embodiment of the present disclosure considers the problem in related technologies where a document cannot be recovered if a cache node fails after caching it. When caching the target document to a local cache cluster, the pages in the target document are pre-divided into multiple blocks along the row and column dimensions, and cached on a block-by-block basis. This enables fine-grained caching of the target document, allowing each block to potentially be cached on different cache nodes. Thus, if a cache node fails, only one block of the target document is unrecoverable, not the entire document. Only a small block of document data needs to be recovered, improving the non-volatility of document data and reducing the difficulty of document data recovery. Furthermore, after obtaining the processing request from the target processing node for the target block in the target document, the target local cache node corresponding to the block index can be determined in the local cache cluster bound to the target processing node based on the block index of the target block contained in the processing request. This allows for quick data querying of the locally deployed local cache cluster of the target processing node, improving data query efficiency. Finally, if it is determined that the target local cache node contains the target chunk, the target chunk is directly retrieved from the local cache node; otherwise, if it is determined that the target local cache does not contain the target chunk, the target chunk is retrieved from the remote data source. This method only transmits the required target chunk when calling cached data, instead of transmitting the entire target document, which can reduce document transmission time in document processing and improve document transmission efficiency.

[0153] The above is a general description of steps 310-340. The following will provide a detailed description of the specific implementation of steps 310-340.

[0154] Step 310 will be described in detail below.

[0155] In step 310, the processing request of the target processing node for the target block in the target document is obtained. The page in the target document is pre-divided into multiple blocks in terms of rows and columns. The multiple blocks include the target block. The processing request contains the block index of the target block.

[0156] Please refer to Figure 4 In one embodiment, the processing request for the target chunk is generated by the target processing node based on the received processing task for the target document; the processing task for the target document is allocated to the target processing node by the task scheduling node in the target task cluster through the following process:

[0157] Step 410: Obtain the current number of tasks and the current node status of each candidate processing node in the target task cluster;

[0158] Step 420: For each candidate processing node, determine the first score based on the current number of tasks;

[0159] Step 430: Determine the second score based on the current node state;

[0160] Step 440: Based on the first score and the second score, determine the total node score of the candidate processing node;

[0161] Step 450: Based on the total node score, determine the target processing node from multiple candidate processing nodes and assign the processing task to the target processing node.

[0162] Steps 410-450 are described in detail below.

[0163] A target task cluster refers to a cluster that can efficiently process large amounts of data. A target task cluster often contains a task scheduling node and multiple candidate processing nodes.

[0164] A task scheduling node is a node in a target task cluster that is responsible for coordinating task scheduling and data acquisition among various candidate processing nodes. The node functions of a task scheduling node include allocating tasks to candidate processing nodes so that the available resources of the candidate processing nodes can be used to execute the tasks.

[0165] Candidate processing nodes refer to cluster nodes that execute tasks assigned by the task scheduling node.

[0166] A processing task targeting a target document refers to a task that processes data from a portion of the document's data. These processing tasks are often generated based on user requests within different business scenarios.

[0167] In step 410, the current number of tasks of the candidate processing node is used to indicate the total number of tasks that the candidate processing node is currently processing.

[0168] The current node status of a candidate processing node is used to indicate the processing delay and resource status of the candidate processing node at the current time.

[0169] In this specific implementation, the working status of each candidate processing node in the target task cluster is recorded in real time by the cluster computing server in the server background. Based on this, with authorization, the task scheduling node can obtain the current number of tasks and the current node status of each candidate processing node from the background logs of the cluster computing server.

[0170] In step 420, the first score is used to statistically indicate the saturation level of the tasks being processed by the candidate processing node.

[0171] In this specific implementation, a function calculation method can be used to calculate the first score based on the current number of tasks. Specifically, firstly, a preset function is called. This preset function is a functional equation indicating how the first score changes with the current number of tasks. The preset function is a decreasing function with the current number of tasks as the independent variable and the first score as the dependent variable. The larger the current number of tasks, the smaller the first score. Next, the current number of tasks of the candidate processing node is input into the preset function, and the output of the preset function is used as the first score.

[0172] In step 430, the second score is used to statistically indicate the quality of the candidate processing node's node state at the current time.

[0173] To save space, the specific process of determining the second score based on the current node state in this embodiment will be described in detail below, and will not be repeated here.

[0174] In step 440, the total node score is used to statistically indicate the suitability of the candidate processing node for processing the target file. Specifically, a higher total node score indicates that the candidate processing node is more suitable as a processing node for processing the target file.

[0175] In the specific implementation of this embodiment, firstly, a first weight and a second weight are determined. The first weight indicates the influence of the current number of tasks of the candidate processing node on node selection; the second weight indicates the influence of the current node state of the candidate processing node on node selection. The sum of the first weight and the second weight is 1. Next, the first weight and the first score are multiplied to obtain a first product result; the second weight and the second score are multiplied to obtain a second product result; the first product result and the second product result are added together to obtain the total node score of the candidate processing node.

[0176] In step 450, based on the total node score, the target processing node is determined from multiple candidate processing nodes, and the processing task is assigned to the target processing node.

[0177] In the specific implementation of this embodiment, when determining the target processing node from multiple candidate processing nodes, a higher total node score indicates a more balanced number of current tasks and a more balanced current node state at the current time, thus better meeting the conditions for assigning processing tasks. Based on this, the candidate processing node with the highest total node score can first be selected as the target processing node to improve the task execution efficiency. Then, the processing tasks are assigned to the target processing node.

[0178] In another embodiment, when determining the target processing node from multiple candidate processing nodes, one can first select candidate processing nodes whose total score is greater than a preset threshold as intermediate processing nodes. Then, one of the intermediate processing nodes is randomly selected as the target processing node, and the processing task is assigned to the target processing node. This method improves the flexibility and freedom of node selection when choosing a processing node whose current task count and current node status both meet the task assignment conditions.

[0179] The advantage of this embodiment is that when the task scheduling node allocates processing tasks, it takes into account both the current number of tasks and the current node status of each candidate processing node. It also quantifies the impact of the current number of tasks and the current node status of each candidate processing node on node selection from a statistical perspective, calculates the total node score of each candidate processing node, and then uses the total node score to select nodes. This helps to select candidate processing nodes with fewer current tasks and better current node status as target processing nodes to execute processing tasks for the target document, thereby improving the rationality of task allocation and improving the efficiency and reliability of task execution.

[0180] In this embodiment of the disclosure, the current node status of the candidate processing node includes the node latency level and the node resource status.

[0181] Node latency is used to indicate the time or speed at which a candidate processing node processes a single task.

[0182] Node resource status is used to indicate how much computing resources a candidate processing node is allowed to access at the current time.

[0183] Please refer to Figure 5 In one embodiment, step 430 specifically includes, but is not limited to, the following steps 510-530:

[0184] Step 510: Determine the first sub-score based on the node latency.

[0185] Step 520: Determine the second sub-score based on the node resource status;

[0186] Step 530: Determine the second score of the candidate processing node based on the weighted sum of the first sub-score and the second sub-score.

[0187] Steps 510-530 are described in detail below.

[0188] In step 510, the first sub-score is used to quantitatively evaluate the node latency of candidate processing nodes from a statistical perspective. Specifically, the greater the node latency, the smaller the first sub-score.

[0189] In the specific implementation of this embodiment, the process of step 510 is similar to that of step 420 described above. To save space, it will not be described again.

[0190] In step 520, the second sub-score is used to quantitatively evaluate the node resource status of the candidate processing node from a statistical perspective. Specifically, the more computing resources available on a candidate processing node, the better its node resource status, and the larger the second sub-score.

[0191] In the specific implementation of this embodiment, the process of step 520 is similar to that of step 420 described above. To save space, it will not be described again.

[0192] In step 530, the second score of the candidate processing node is determined based on the weighted sum of the first sub-score and the second sub-score.

[0193] In the specific implementation of this embodiment, the process of step 530 is similar to that of step 440 described above. To save space, it will not be described again.

[0194] like Figure 6A The diagram illustrates the calculation of the second score for each candidate processing node. Specifically, for candidate processing node 1 with a node latency of 20ms and a node resource status (available computing resources) of 100, the first sub-score is 80 and the second sub-score is 80; for candidate processing node 2 with a node latency of 30ms and a node resource status (available computing resources) of 150, the first sub-score is 70 and the second sub-score is 85; for candidate processing node 3 with a node latency of 25ms and a node resource status (available computing resources) of 80, the first sub-score is 75 and the second sub-score is 78; and for candidate processing node 1 with a node latency of 40ms and a node resource status (available computing resources) of 200, the first sub-score is 60 and the second sub-score is 90. Based on this, when the weights of the first and second sub-scores are both 0.5, the second score of candidate processing node 1 is 80, the second score of candidate processing node 2 is 77.5, the second score of candidate processing node 3 is 76.5, and the second score of candidate processing node 4 is 75.

[0195] like Figure 6BThe diagram illustrates the calculation of the total score for each candidate processing node. Specifically, candidate processing node 1 has 2 current tasks, so its first score is 90; candidate processing node 2 has 4 current tasks, so its first score is 80; candidate processing node 3 has 5 current tasks, so its first score is 75; and candidate processing node 4 has 8 current tasks, so its first score is 60. Based on this, when both the first and second weights are 0.5, the total score for candidate processing node 1 is 85, for candidate processing node 2 it is 78.75, for candidate processing node 3 it is 75.75, and for candidate processing node 4 it is 67.5. Therefore, candidate processing node 1 is chosen as the target processing node to handle the processing tasks for the target document.

[0196] The advantage of this embodiment is that, when calculating the second score for each candidate processing node, it considers analyzing the current node state from multiple dimensions, such as node latency and node resource status. Therefore, by statistically quantifying the impact of node latency and node resource status on node state analysis based on the candidate processing node's latency and resource status, and calculating the second score for each candidate processing node, a comprehensive consideration of the factors influencing node state can be achieved, thereby improving the accuracy of score calculation.

[0197] The following provides an example description of steps 320-340.

[0198] In steps 320-340, the target local cache node corresponding to the block index is determined in the local cache cluster bound to the target processing node. If the target local cache node contains the target block, the target block is obtained from the target local cache node; if the target local cache does not contain the target block, the target block is obtained from the data source.

[0199] like Figure 7A The diagram illustrates a simplified workflow for cached data processing. Specifically, when the data processing engine or the Spark Engine needs to access document data during various computational tasks, it first retrieves the relevant document data from the local fine-grained cache. Only when the relevant document data is not present in the local fine-grained cache does it retrieve the document data from the remote storage, cache it in the local cache, and then transmit it to the data processing engine or the Spark Engine.

[0200] like Figure 7B The diagram illustrates the specific implementation process of invoking cached data during task execution. Specifically, when a processing task for a target document needs to be executed, the cluster computing system includes a task scheduling node (Spark Driver) and several candidate processing nodes (Spark Executors). The task scheduling node selects candidate processing node 1 as the target processing node from among all candidate processing nodes 1, 2, and 3, and assigns the processing task for the target file, thus achieving distributed task scheduling. Next, when candidate processing node 1 executes the processing task for the target document, it first requests cached data related to the processing task (e.g., the target chunk of the document) from the local cache cluster (Data United Cache). Further, based on the request sent by the target processing node (candidate processing node 1), it determines whether a local cache node in the local cache cluster has been hit. If a local cache node corresponding to the chunk index of the cached data exists in the local cache cluster, it is determined that a local cache node in the local cache cluster has been hit, and the cached data is retrieved from the cache pool of the local cache cluster and fed back to candidate processing node 1. If no local cache node corresponding to the block index of the cached data exists in the local cache cluster, it is determined that the local cache node was not found, and the Triger Data loading process is triggered to automatically load the data into the local cache cluster (Cachedata loading plugin). Data is retrieved from a remote data source, which stores the data in columnar storage format Parquet or columnar storage format ORC. Further, when data related to the processing task is found in the remote data source, the data is uploaded to the local cache cluster. After the data is saved to the cache pool of the local cache cluster, the cached data is retrieved and fed back to candidate processing node 1. In addition, a corresponding data update mechanism can be set in the cache pool of the local cache cluster to evict and update various cached data. The local cache cluster can be a self-built distributed cluster.

[0201] The advantage of this embodiment is that, during task execution, the task scheduling node uniformly schedules and allocates each task, which helps improve the rationality of task allocation. Furthermore, after a candidate processing node receives an allocated task, it first checks if a local cache node is matched in the local cache cluster. If a local cache node is matched, cached data can be directly retrieved from the local cache node. Only if a local cache node is not matched in the local cache cluster will document data be retrieved from a remote data source. After caching the document data, the cached data required by the task is then fed back to the candidate processing node. This method reduces document transmission time during document processing and improves document transmission efficiency. In addition, since the local cache cluster is bound to the target processing node, storing cached data near the computing node (target processing node) achieves cache locality, allowing access to cached data to be implemented in the local cache. This helps reduce network transmission distance and time, improving data access speed.

[0202] The process of pre-storing frequently used data in a local cache cluster according to one embodiment of the present disclosure is described in detail below.

[0203] Please refer to Figure 8 In one embodiment, before obtaining the processing request from the target processing node for the target chunk in the target document, the cached data processing method further includes, but is not limited to, the following steps 810-820:

[0204] Step 810: Determine the data retrieval frequency and data size of multiple reference data;

[0205] Step 820: Based on the data call frequency and data size, select candidate data from multiple reference data and cache the candidate data in the local cache cluster.

[0206] Steps 810-820 are described in detail below.

[0207] In step 810, the data retrieval frequency and data size of multiple reference data are determined.

[0208] Reference data refers to data that can be provided by various data sources, is frequently used, and is not often modified.

[0209] Data retrieval frequency is used to indicate the number of times reference data will be retrieved within a fixed time period.

[0210] Data size is used to indicate the amount of data that the reference data has. For example, data size can refer to the number of bytes in the reference data.

[0211] In the specific implementation of this embodiment, since the data call frequency and data size of each reference data are often recorded in the server corresponding to the data source where the reference data is located, based on this, with authorization, the data call frequency and data size of the reference data can be retrieved from the background log of the server corresponding to the data source where the reference data is located.

[0212] In step 820, candidate data is determined from multiple reference data based on data call frequency and data size, and the candidate data is cached in the local cache cluster.

[0213] Alternate data refers to the specific data that is ultimately selected to be cached in the local cache cluster before the target processing node executes the task.

[0214] In a specific implementation of this embodiment, step 820 may include, but is not limited to, the following steps:

[0215] The third score is determined based on the frequency of data access.

[0216] The fourth score is determined based on the size of the data.

[0217] The data scores for the reference data are determined based on the third and fourth scores;

[0218] Based on the data scores, alternative data are determined from multiple reference data, and the alternative data is cached in the local cache cluster.

[0219] Specifically, the process of determining the third score based on data call frequency and the process of determining the fourth score based on data size are similar to step 420 above. The process of determining the data score of the reference data based on the third and fourth scores is similar to step 440 above. The process of determining candidate data from multiple reference data based on the data scores and caching the candidate data in the local cache cluster is similar to step 450 above. To save space, these will not be elaborated further.

[0220] It should be noted that the specific process of caching candidate data into the local cache cluster in this embodiment of the present disclosure is basically the same as the specific process of caching each block of the target document into the local cache cluster. Since the specific process of caching each block of the target document into the local cache cluster will be described in detail below, it will not be repeated here.

[0221] like Figure 9The diagram illustrates the calculation of data scores for each reference data point. Specifically, for reference data 1, with a data access frequency of 20, a data size of 100 bytes, and originating from data source 1, the third score is 60 and the fourth score is 80; for reference data 2, with a data access frequency of 30, a data size of 150 bytes, and originating from data source 1, the third score is 65 and the fourth score is 85; for reference data 3, with a data access frequency of 25, a data size of 80 bytes, and originating from data source 2, the third score is 70 and the fourth score is 92; and for reference data 4, with a data access frequency of 40, a data size of 200 bytes, and originating from data source 3, the third score is 80 and the fourth score is 80. Based on this, when the weights of the third and fourth scores are both 0.5, the data score for reference data 1 is 75, the data score for reference data 2 is 75, the data score for reference data 3 is 81, and the data score for reference data 4 is 80. Based on this, reference data 3 and reference data 4, which have higher data scores, are used as alternative data and are pre-stored in the local cache cluster.

[0222] The advantage of this embodiment is that it sets up an active caching mechanism in the local cache cluster, enabling the local cache cluster to proactively pre-store frequently used data from various data sources after its creation. This proactive data preheating avoids waiting until a data retrieval request is received before retrieving and caching basic data, allowing the caching of frequently used data to be performed in advance, thereby improving the efficiency of cache data retrieval and task execution. This active caching method can preheat and adjust cache content, which is beneficial for adapting to different load conditions, improving cache hit rate, and reducing performance issues caused by dynamic load changes.

[0223] The following is a detailed description of an embodiment of the present disclosure of caching target documents to a local cache cluster.

[0224] Please refer to Figure 10 In one embodiment, before obtaining the processing request from the target processing node for the target chunk in the target document, the cached data processing method further includes, but is not limited to, the following steps 1010-1030:

[0225] Step 1010: Divide the pages in the target document into multiple blocks along the dimensions of rows and columns;

[0226] Step 1020: Determine the block index based on the block's digest value;

[0227] Step 1030: Store the blocks in the local cache node corresponding to the block index.

[0228] Steps 1010-1030 are described in detail below.

[0229] In step 1010, the pages in the target document are divided into multiple blocks in terms of rows and columns.

[0230] In this specific implementation, firstly, the pages in the target document are divided into multiple row groups along the row dimension, where each row group contains one or more rows of data. Next, each row group is divided into multiple column blocks along the column dimension, with each column block serving as a separate block, resulting in multiple blocks. Each block includes one or more columns of data. This allows the data of a file (e.g., a document) to be divided into smaller units, and a high-efficiency data structure is used for cache management at the small unit (e.g., block) level, achieving fine-grained caching.

[0231] like Figure 11A The diagram illustrates how a document is divided into multiple chunks along the dimensions of rows and columns. Specifically, a document contains multiple rows of data, with the footer containing metadata describing the document's structure, pages, and other details. The document is first divided into multiple row groups, each containing multiple rows. Next, each row group is further divided into multiple column chunks, where each chunk contains the document content of a specific column from a specific row.

[0232] like Figure 11B The diagram illustrates how a document is divided into multiple blocks along the dimensions of rows and columns. Specifically, when dividing a document along the dimensions of rows and columns, the document is divided into 20 blocks of equal size. When divided along the row dimension, the document is divided into 5 row groups. Each row group is then divided into 4 column blocks, resulting in 20 blocks.

[0233] In steps 1020-1030, the block index is determined based on the block's digest value. The block is then stored in the local cache node corresponding to the block index.

[0234] The digest value of a block refers to the result obtained by performing a digest operation on the specific data in the block using a predetermined digest algorithm. The digest value of a block is often a random string of fixed length.

[0235] To save space, the specific process of determining the block index and caching the blocks to the local cache node in this embodiment will be described in detail below, and will not be repeated here.

[0236] The advantage of this embodiment is that, when caching the target document, it does not cache the entire target document as a whole, but rather divides the target document into multiple blocks along the row and column dimensions, and caches them on a block-by-block basis, transforming coarse-grained caching into fine-grained caching. Furthermore, when caching a single block, the block index is determined based on the block's digest value, and the block is stored in the corresponding local cache node according to the mapping relationship between the block index and each local cache node, which improves the standardization and rationality of block caching.

[0237] It should be noted that when cached data is cached in a local cache cluster, a multi-level caching approach is adopted, caching data at different levels to reduce the number of accesses to the underlying storage system (remote data source), thereby reducing latency and improving throughput. Specifically, the multi-level caching in this embodiment includes in-process caching, node caching, and distributed caching. In-process caching is used for frequently accessed cached data, caching it using the memory space of a single process. Node caching caches data in the memory of a single local cache node, ensuring that the cached data is shared by all processes corresponding to the local cache node, improving the utilization rate of cached data. Distributed caching shares cached data among multiple local cache nodes in the local cache cluster, performing distributed caching based on the cached data's summary information.

[0238] In one embodiment, the specific process of determining the block index based on the block's digest value includes:

[0239] Determine the remainder of the summary value modulo the first number, and use it as the block index.

[0240] The first number is the number of local cache nodes in the local cache cluster.

[0241] Specifically, firstly, a predefined digest algorithm is used to perform a digest operation on the specific data in the block, obtaining the digest value of the block. Next, the number of local cache nodes in the local cache cluster is counted to obtain a first count. Further, the digest value is modulo the first count to obtain a remainder, which is then used as the block index. This method simplifies the representation of the block index.

[0242] In this embodiment, the specific process of storing the blocks into the local cache node corresponding to the block index includes:

[0243] The blocks are stored in local cache nodes with the same sequence number as the block index.

[0244] Specifically, first, the node sequence number assigned to each local cache node in the local cache cluster is obtained. The node sequence number can be set according to the total number of local cache nodes in the local cache cluster. Next, based on the node sequence number, the local cache node whose node sequence number matches the block index is found, and the block is cached in the found local cache node.

[0245] like Figure 12 The diagram illustrates a specific example of storing blocks in the local cache nodes corresponding to the block index. Specifically, each local cache node is assigned a node number; for example, the node number of local cache node 1 is set to 1, the node number of local cache node 2 is set to 2, and so on. Based on this, when the remainder of the summary value of a block of the target document modulo the first number of all local cache nodes is 13, the block is cached on local cache node 13 with node number 13.

[0246] The advantage of this embodiment is that when determining the block index of the local cache node, the digest value of the block and the total number of local cache nodes are taken into account, so that the block index can be represented by a single value, thereby simplifying the representation of the block index. At the same time, storing the block in the local cache node with the same sequence number as the block index can achieve reasonable caching of the block and make the cached data of each local cache node more balanced.

[0247] In another embodiment, the specific process of determining the block index based on the block's digest value includes:

[0248] The summary value is determined as the block index.

[0249] Specifically, after performing a digest operation on the specific data in the blocks to obtain the digest value, the digest value is directly used as the block index, thereby simplifying the process of determining the block index.

[0250] In this embodiment, the local cache node has a block index range, and the specific process of storing blocks into the local cache node corresponding to the block index includes:

[0251] The blocks are stored in the local cache node corresponding to the block index range where the block index is located.

[0252] The block index range indicates the specific range of digest values ​​corresponding to the block index of cached data that the local cache node is allowed to cache.

[0253] Specifically, first, the block index range where the block index is located is determined by comparing the block index range of each local cache node with the block index itself. Then, the block is stored in the local cache node corresponding to the block index range where the block index is located.

[0254] The advantage of this embodiment is that, when determining the block index of a local cache node, directly using the block digest value as the block index improves the efficiency of block index determination. Simultaneously, pre-allocating a block index range for each local cache node and storing the blocks in the local cache node corresponding to the block index range where the block index is located improves the accuracy of block caching.

[0255] Please refer to Figure 13 In one embodiment, the specific process of determining the block index range of a local cache node may include, but is not limited to, the following steps 1310-1340:

[0256] Step 1310: Obtain the preset summary data body;

[0257] Step 1320: Determine the total number of cache nodes in the local cache cluster;

[0258] Step 1330: Based on the total number of cache nodes, divide the digest data body into equal parts to obtain multiple digest data segments;

[0259] Step 1340: Match multiple local cache nodes with multiple digest data segments so that each local cache node corresponds to a digest data segment, and use the digest data segment as the block index range corresponding to the local cache node.

[0260] Steps 1310-1340 are described in detail below.

[0261] In step 1310, a preset summary data body is obtained.

[0262] The summary data body is a ring-shaped space with multiple slots, each slot corresponding to a summary value.

[0263] In this specific implementation, firstly, a series of digest values ​​are determined. Then, this series of digest values ​​is mapped to a ring space to obtain the digest data volume.

[0264] For example, a hash space can be mapped to a virtual ring, and the virtual ring can be used as the digest data body, where the hash space takes values ​​ranging from 0 to (2). 32 -1).

[0265] In step 1320, the total number of cache nodes in the local cache cluster is determined.

[0266] The total number of cache nodes indicates the total number of local cache nodes in the local cache cluster.

[0267] In the specific implementation of this embodiment, the summation function or similar methods can be used to count the number of local cache nodes in the local cache cluster to obtain the total number of cache nodes.

[0268] In step 1330, based on the total number of cache nodes, the digest data body is divided into multiple digest data segments.

[0269] A summary data segment is a data segment consisting of multiple consecutive summary values.

[0270] In this specific implementation, the digest data body is divided equally according to the total number of cache nodes to obtain a total number of digest data segments.

[0271] In step 1340, multiple local cache nodes are matched with multiple digest data segments so that each local cache node corresponds to a digest data segment, and the digest data segment is used as the block index range corresponding to the local cache node.

[0272] In this specific implementation, when matching multiple local cache nodes with multiple digest data segments, the local cache nodes and digest data segments are first randomly matched so that each local cache node corresponds to one digest data segment, and the digest data segments matched by different local cache nodes are different. Then, for each local cache node, the matched digest data segment is used as the block index range corresponding to that local cache node.

[0273] like Figure 14A The diagram illustrates the specific steps for setting block index intervals for each local cache node. Specifically, for a circular digest data body, when there are four local cache nodes in the local cache cluster, the circular digest data body is divided into four equal parts, resulting in four digest data segments of the same size. Further, the digest data segment in the upper left corner is designated as block index interval 1 for local cache node 1; the digest data segment in the lower left corner is designated as block index interval 3 for local cache node 3; the digest data segment in the upper right corner is designated as block index interval 2 for local cache node 2; and the digest data segment in the lower right corner is designated as block index interval 4 for local cache node 4.

[0274] like Figure 14B The diagram illustrates a specific instance of storing blocks in the local cache node corresponding to the block index. Specifically, for a single block whose block index is in block index range 1, the mapping position of this block in the digest data body is within the range of block index range 1. Therefore, this single block is cached on local cache node 1. Similarly, for a single block whose block index is in block index range 3, the mapping position of this block in the digest data body is within the range of block index range 3. Therefore, this single block is cached on local cache node 3.

[0275] The advantage of this embodiment is that by dividing a circular digest data body equally according to the total number of local cache nodes in the local cache cluster, multiple digest data segments with equal data volume are obtained. This enables the uniform distribution of digest data segments, ensuring that the digest values ​​contained in the block index intervals of each local cache node are equal. This improves the rationality of determining the block index intervals, which is conducive to realizing distributed caching of cached data and improving the load balancing of each local cache node.

[0276] Please refer to Figure 15 In another embodiment, the specific process of determining the block index based on the block's digest value includes:

[0277] The summary value is determined as the block index.

[0278] In this embodiment, step 1030 specifically includes, but is not limited to, the following steps 1510-1530:

[0279] Step 1510: Determine the virtual digest node corresponding to each local cache node;

[0280] Step 1520: For each virtual summary node, calculate the difference between the block index and the node summary information to obtain the summary difference;

[0281] Step 1530: Based on the digest difference, determine the local cache node corresponding to the block index and store the blocks in the local cache node corresponding to the block index.

[0282] Steps 1510-1530 are described in detail below.

[0283] In step 1510, the virtual digest node corresponding to each local cache node is determined.

[0284] Each virtual summary node has node summary information, and each virtual summary node is located on the same preset summary data body.

[0285] A virtual digest node refers to a virtual node that a local cache node maps to on the digest data body.

[0286] The node summary information is used to indicate the summary value corresponding to the local cache node in the summary data body.

[0287] In this specific implementation, firstly, based on the first number of local cache nodes in the local cache cluster, a first number of slot positions containing digest values ​​are randomly selected on the digest data body. Next, a virtual digest node is set at each selected slot position, resulting in the first number of virtual digest nodes. Further, local cache nodes and virtual digest nodes are randomly mapped, so that each local cache node is mapped to a virtual digest node, resulting in virtual digest nodes corresponding to each local cache node. The digest value corresponding to the slot position of each virtual digest node is then used as the node digest information of that virtual digest node (local cache node).

[0288] In step 1520, for each virtual summary node, the difference between the block index and the node summary information is calculated to obtain the summary difference.

[0289] The summary difference is used to indicate the degree of difference between the block index and the node summary information.

[0290] In the specific implementation of this embodiment, for each virtual digest node, the difference between the digest value corresponding to the block index and the digest value in the node digest information is calculated to obtain the digest difference.

[0291] In addition, the mapping position of the block can be located on the preset summary data body according to the summary value corresponding to the block index, and the summary difference can be calculated according to the difference between the mapping position and the position of each virtual summary node on the preset summary data body.

[0292] In step 1530, based on the digest difference, the local cache node corresponding to the block index is determined, and the block is stored in the local cache node corresponding to the block index.

[0293] In this specific implementation, since the smaller the digest difference, the closer the mapping position of the block on the digest data body is to the position of the virtual digest node of the local cache node, the more effective the implementation. Therefore, firstly, based on the digest difference, the local cache node with the smallest digest difference is determined as the local cache node corresponding to the block index. Next, the block is stored in the local cache node corresponding to the block index.

[0294] like Figure 16The diagram illustrates a specific instance of storing blocks in the local cache nodes corresponding to their block indices. Specifically, a predefined digest data body contains virtual digest nodes corresponding to local cache node 4, local cache node 1, local cache node 2, and local cache node 3. Based on this, since the block indices of blocks 5 and 3 are closer to the virtual nodes corresponding to local cache node 1 in a clockwise direction, blocks 5 and 3 are cached in local cache node 1; the block index of block 6 is closer to the virtual node corresponding to local cache node 1, so block 6 is cached in local cache node 2; the block indices of blocks 1 and 2 are closer to the virtual nodes corresponding to local cache node 3, so blocks 1 and 2 are cached in local cache node 3; and the block index of block 4 is closer to the virtual node corresponding to local cache node 4, so block 4 is cached in local cache node 4.

[0295] The advantage of this embodiment is that, based on the number of local cache nodes in the local cache cluster, multiple virtual digest nodes are constructed on the digest data body, and a mapping relationship is established between each local cache node and the virtual digest nodes. Furthermore, the mapping position of the block index on the digest data body and the positional relationship of each virtual digest node are quantified using the digest difference. Blocks are preferentially cached on local cache nodes whose block indexes are closer to the virtual digest nodes, enabling distributed caching of each block. This ensures that all data requiring caching is cached evenly, reducing situations where some local cache nodes are overloaded due to excessive cached data, or underloaded due to insufficient cached data, thus improving the overall cache space utilization efficiency of the local cache cluster. Additionally, since local cache nodes and each block (cache data) are mapped to the same virtual digest data body, specific locations can be determined based on the digest value of the cached data, facilitating distributed caching of cached data.

[0296] The following is a detailed description of the data processing procedure when a node change occurs in a local cache cluster according to an embodiment of this disclosure.

[0297] Please refer to Figure 17 In one embodiment, the data processing procedure when a node exits in the local cache cluster may include, but is not limited to, the following steps 1710-1730:

[0298] Step 1710: If the target local cache node is detected to have exited the local cache cluster, then clear the virtual digest node corresponding to the target local cache node on the digest data body, and determine the digest value of each block in the target local cache node.

[0299] Step 1720: Based on the summary value, locate the blocks on the summary data body to obtain the data positions of the blocks;

[0300] Step 1730: Based on the data location, restore the blocks to the local cache node corresponding to the virtual digest node whose distance from the data location meets the first condition.

[0301] Steps 1710-1730 are described in detail below.

[0302] In step 1710, if it is detected that the target local cache node has exited the local cache cluster, the virtual digest node corresponding to the target local cache node is cleared on the digest data body, and the digest value of each block in the target local cache node is determined.

[0303] In the specific implementation of this embodiment, if it is detected that the target local cache node has exited the local cache cluster, it indicates that the target local cache node is no longer suitable for caching any data. Therefore, the virtual digest node corresponding to the target local cache node is first cleared from the digest data body to make the virtual digest node on the digest data body consistent with the actual situation of the local cache cluster. Next, for each block of the target local cache node, the digest value of each block is determined.

[0304] In step 1720, based on the digest value, blocks are located on the digest data body to obtain the data location of the blocks.

[0305] Data location is used to indicate the specific location where the chunk should be placed on the summary data body.

[0306] In the specific implementation of this embodiment, the blocks are mapped to the summary data body according to the summary value to achieve the positioning of the blocks; since the summary data body is composed of multiple slots containing summary values, the slot position containing the summary value of the block is used as the data position of the block on the summary data body.

[0307] In step 1730, based on the data location, the blocks are re-stored on the local cache node corresponding to the virtual digest node whose distance from the data location meets the first condition.

[0308] The distance parameter is used to indicate the degree of difference in position between the digest value of the block and the virtual digest node corresponding to the local cache node on the preset digest data body.

[0309] The first condition is used to limit the relative position requirements between the block and the local cache node to be cached in the preset digest data body.

[0310] In this specific implementation, firstly, the distance from the data location of the block to each virtual digest node is calculated in a clockwise direction to obtain the block distance. Then, among the multiple virtual digest nodes, the local cache node corresponding to the virtual digest node with the smallest block distance (the virtual digest node closest to the block's block index) is selected to cache the block.

[0311] The advantage of this embodiment is that it takes into account the scenario where cached data needs to be migrated when the target local cache node leaves the local cache cluster. For each block of the target local cache node, based on the distance between the block index and each virtual digest node on the digest data body in the clockwise direction, the block is preferentially cached on the local cache node with the smallest distance, which can realize the rapid migration of cached data and improve the cache security and stability of cached data.

[0312] Please refer to Figure 18 In one embodiment, the data processing procedure when a new node is added to the local cache cluster may include, but is not limited to, the following steps 1810-1840:

[0313] Step 1810: If a new cache node is detected in the local cache cluster, determine the new summary node corresponding to the new cache node in the summary data body;

[0314] Step 1820: Determine the node distance between the newly added summary node and each virtual summary node;

[0315] Step 1830: Based on node distance, determine the target digest node among multiple virtual digest nodes, and determine the local cache node corresponding to the target digest node as the base cache node;

[0316] Step 1840: Redistribute the multiple blocks in the baseline cache node between the new cache node and the baseline cache node.

[0317] Steps 1810-1840 are described in detail below.

[0318] In step 1810, if a new cache node is detected in the local cache cluster, the new summary node corresponding to the new cache node is determined in the summary data body.

[0319] In the specific implementation of this embodiment, if a new cache node is detected in the local cache cluster, a new cache node is generated by randomly selecting a slot position other than the virtual cache node on the digest data body, and the mapping relationship between the new cache node and the new cache node is determined.

[0320] In step 1820, the node distance between the newly added summary node and each virtual summary node is determined.

[0321] Node distance is used to indicate the degree of positional difference between newly added summary nodes and virtual summary nodes on the preset summary data body.

[0322] In this specific implementation, the location of the newly added summary node is located on the preset summary data body, and the node distance between the newly added summary node and each virtual summary node is calculated based on the difference between their locations on the preset summary data body and the locations of each virtual summary node. The node distance can be represented by the number of slots the newly added summary node must traverse clockwise to reach each virtual summary node.

[0323] In step 1830, based on node distance, the target digest node is determined among multiple virtual digest nodes, and the local cache node corresponding to the target digest node is determined as the base cache node.

[0324] In the specific implementation of this embodiment, firstly, based on node distance, the virtual digest node with the smallest node distance is determined as the target digest node. Next, according to the correspondence between each local cache node and the virtual digest node, the local cache node corresponding to the target digest node is determined as the base cache node.

[0325] In step 1840, multiple blocks in the baseline cache node are redistributed between the new cache node and the baseline cache node.

[0326] In a specific implementation of this embodiment, step 1840 may include, but is not limited to, the following steps:

[0327] Determine the first cache size of the newly added cache node, the second cache size of the baseline cache node, and the number of blocks in the partition;

[0328] The first ratio is determined based on the first cache size and the second cache size;

[0329] Based on the first ratio and the number of blocks, the blocks are redistributed among the new cache nodes and the baseline cache nodes.

[0330] The first cache size indicates the total amount of data that a newly added cache node is allowed to cache, and the second cache size indicates the total amount of data that the baseline cache node is allowed to cache. The number of blocks indicates the total number of blocks already cached by the baseline cache node. The first ratio is determined based on the ratio of the first cache size to the second cache size.

[0331] Specifically, firstly, the amount of cached data that the new cache node and the base cache node can accommodate is statistically analyzed to obtain the first cache size of the new cache node and the second cache size of the base cache node. The number of blocks in the base cache node is then counted to obtain the number of blocks. Next, the first cache size is divided by the second cache size to obtain a first ratio. Further, based on the first ratio and the number of blocks, the blocks are divided to obtain a first sub-number of blocks to be cached in the new cache node and a second sub-number of blocks to be cached in the base cache node. The sum of the first and second sub-numbers is the total number of blocks. Finally, the first sub-number of blocks are cached in the new cache node, and the second sub-number of blocks are cached in the base cache node, wherein the sum of the data sizes of the first sub-number of blocks is not greater than the first cache size.

[0332] The advantage of this embodiment is that it takes into account the situation where a portion of cached data needs to be migrated when a new local cache node is added to the local cache cluster. For the local cache node that is closest to the new cache node in terms of the digest data body, a portion of the cached data of the local cache node (base cache node) is migrated to the new cache node. This can effectively alleviate the cache pressure on the original local cache node when a node is added, and improve the load balance of each local cache node in the local cache cluster.

[0333] Please refer to Figure 19 In one embodiment, the data processing procedure when a node failure occurs in the local cache cluster may include, but is not limited to, the following steps 1910-1930:

[0334] Step 1910: If a node failure is detected in the local cache cluster, locate the virtual digest node corresponding to the failed cache node in the digest data body to obtain the target node location of the virtual digest node corresponding to the failed cache node.

[0335] Step 1920: Based on the target node location and the node location of the virtual digest node corresponding to each local cache node, determine the first cache node among multiple local cache nodes;

[0336] Step 1930: Redistribute the blocks in the faulty cache node among multiple first cache nodes.

[0337] Steps 1910-1930 are described in detail below.

[0338] In step 1910, if a node failure is detected in the local cache cluster, the virtual digest node corresponding to the failed cache node is located on the digest data body to obtain the target node location of the virtual digest node corresponding to the failed cache node.

[0339] A faulty cache node refers to a local cache node in a local cache cluster that is unable to function properly.

[0340] The target node location is used to indicate the specific location of the virtual digest node corresponding to the fault cache node in the digest data body.

[0341] In the specific implementation of this embodiment, if a node failure is detected in the local cache cluster, the virtual digest node corresponding to the failed cache node is located on the digest data body according to the mapping relationship between each local cache node and the virtual digest node, and the slot position where the virtual digest node is located is determined as the target node position of the virtual digest node corresponding to the failed cache node.

[0342] In step 1920, the first cache node is determined among multiple local cache nodes based on the target node location and the node location of the virtual digest node corresponding to each local cache node.

[0343] In the specific implementation of this embodiment, based on the target node position and the node position of the virtual digest node corresponding to each local cache node, the local cache node corresponding to the virtual digest node at the second number of node positions closest to the target node position in the clockwise direction is taken as the first cache node. The second number can be set according to actual business needs and is not limited.

[0344] In step 1930, the blocks in the faulty cache node are redistributed among multiple first cache nodes.

[0345] In a specific implementation of this embodiment, step 1930 may include, but is not limited to, the following steps:

[0346] Determine the cache node status of each first cache node and the block information of the blocks in the failed cache node;

[0347] Based on the cache node status and block information, a second cache node is selected from multiple first cache nodes, and the blocks from the faulty cache node are re-stored into the second cache node.

[0348] The cache node status indicates the running status of the first cache node, including node latency and the amount of data allowed to be cached. The block information indicates the size of the data blocks that need to be re-cached.

[0349] Specifically, firstly, based on a comparison between the allowed cached data size in the cache node status and the data size in the block information, the first cache node with an allowed cached data size not less than the sum of the data sizes of all blocks to be re-cached is selected as an intermediate cache node. Next, among the multiple intermediate cache nodes, the intermediate cache node with the lowest node latency is selected as the second cache node, and the blocks from the failed cache node are re-stored in the second cache node.

[0350] Furthermore, in order to ensure that the local cache cluster maintains load balancing after all blocks from the failed cache node are re-cached, the specific process of redistributing the blocks from the failed cache node among multiple first cache nodes can be similar to the specific process of redistributing blocks in step 1840. For the sake of brevity, this will not be elaborated further.

[0351] The advantage of this embodiment is that it considers the scenario where cached data needs to be redistributed when a node fails in the local cache cluster. Based on the location of the virtual digest node corresponding to the failed cache node, it prioritizes selecting multiple local cache nodes closer to the failed cache node as first cache nodes. Then, based on the node operating status of each first cache node and the block information of each block to be re-cached, it re-caches each block, effectively reducing the risk of cached data loss and improving the integrity and non-volatility of the cached data in the local cache cluster. The decentralized local cache cluster based on consistent digests in this embodiment effectively avoids single points of failure, improves the fault tolerance of the entire cache system, and enhances the overall availability and stability of the cache system. Because the cached data is more evenly distributed in the local cache cluster, even if the number of local cache nodes changes, the distribution of cached data can still remain relatively balanced, thereby avoiding the problem of load concentration on a few nodes.

[0352] The following is a detailed description of the process of updating cached data in a local cache cluster according to an embodiment of this disclosure.

[0353] Since the cache space of a local cache cluster is often limited, and the usage patterns of different blocks (cache data) often differ, allowing infrequently used or rarely used blocks to be cached in the local cache cluster for extended periods can lead to inefficient cache space utilization and reduced cache cluster efficiency. Therefore, this disclosure provides a scheme for eviction of cached data based on the number of blocks on a local cache node. This scheme enables timely updates to the cached data in the local cache cluster, improving cache space utilization efficiency.

[0354] Please refer to Figure 20In one embodiment, the specific process of updating the cached data of the local cache cluster may include, but is not limited to, the following steps 2010-2020:

[0355] Step 2010: For each block in each local cache node, determine the timestamp of the block being invoked;

[0356] Step 2020: If the number of blocks in the local cache node reaches the second condition, then based on the called timestamp, determine the blocks to be cleared among multiple blocks, and clear the blocks to be cleared in the local cache cluster.

[0357] The following is a detailed description of steps 2010-2020.

[0358] In step 2010, the timestamp of the block being invoked is determined for each block in each local cache node.

[0359] The call timestamp is used to indicate the specific time when the chunk was last called.

[0360] In this specific implementation, the cache server of the local cache cluster records the specific time point when each block on each local cache node in the local cache cluster is invoked each time. Based on this, with authorization, the time point when each block was most recently invoked can be retrieved from the background logs of the cache server of the local cache cluster, thus obtaining the invocation timestamp of the block.

[0361] In step 2020, if it is determined that the number of blocks in the local cache node reaches the second condition, then based on the called timestamp, the block to be cleared is determined from multiple blocks, and the block to be cleared is cleared in the local cache cluster.

[0362] The second condition is used to determine whether the cached data on each local cache node in the local cache cluster has reached the requirement to trigger data clearing.

[0363] Blocks to be cleared refer to blocks whose timestamps do not meet the requirements and need to be eliminated.

[0364] In this specific implementation, firstly, for each local cache node in the local cache cluster, the number of cached blocks on the local cache node is counted to obtain the number of blocks on the local cache node. Next, the number of blocks is compared with the block threshold defined in the second condition. If the number of blocks is greater than or equal to the block threshold, it indicates that there are too many cached blocks on the local cache node, and the number of blocks in the local cache node has reached the second condition, requiring data clearing of the blocks on the local cache node. Based on this, based on the invocation timestamp of each block, a predetermined number of blocks with later invocation timestamps are selected as blocks to be cleared, and these blocks are cleared in the local cache cluster. The block threshold is used to limit the minimum number of blocks required to trigger data clearing, and the predetermined number indicates the number of blocks to be cleared in a single cache data eviction.

[0365] Furthermore, if the number of blocks is less than the block threshold, it indicates that there are fewer blocks cached on the local cache node, and the number of blocks in the local cache node has not met the second condition. Therefore, it is not necessary to clear the data on the blocks on the local cache node.

[0366] like Figure 22A The diagram illustrates the process of evicting blocks from local cache node K based on their call timestamps. Specifically, local cache node K caches eight blocks: block 1 (call timestamp 21:27), block 21 (call timestamp 18:46), block 3 (call timestamp 14:07), block 14 (call timestamp 13:21), block 23 (call timestamp 11:03), block 6 (call timestamp 09:24), block 8 (call timestamp 07:58), and block 17 (call timestamp 07:12). Therefore, when evicting the three blocks with the earliest call timestamps from local cache node K, blocks 6, 8, and 17 are cleared.

[0367] The advantage of this embodiment is that it takes into account that if a certain block (cache data) has been recently accessed, it is more likely to be accessed again in the future. Based on the access timestamps of each block, it can identify the blocks that have been accessed recently and the blocks that have not been accessed recently. Therefore, when the cache data eviction mechanism is triggered, the least recently accessed blocks are cleared. This method can update the cache data of the local cache cluster in a timely manner and clear the cache data that has not been accessed for the longest time, thereby improving the cache space utilization efficiency of the local cache cluster.

[0368] Please refer to Figure 21 In one embodiment, the specific process of updating the cached data of the local cache cluster may include, but is not limited to, the following steps 2110-2120:

[0369] Step 2110: For each block in each local cache node, determine the frequency of the block being called;

[0370] Step 2120: If the number of blocks in the local cache node reaches the second condition, then based on the frequency of being called, determine the blocks to be cleared from among the multiple blocks, and clear the blocks to be cleared in the local cache cluster.

[0371] Steps 2110-2120 are described in detail below.

[0372] In step 2110, the frequency of each block being called is determined for each block in each local cache node.

[0373] The call frequency indicates the total number of times a chunk has been called before the current time.

[0374] In this specific implementation, the cache server of the local cache cluster continuously accumulates the number of times each block on each local cache node in the local cache cluster is called. Based on this, with authorization, the total number of times each block was called before the current time can be retrieved from the background log of the cache server of the local cache cluster, thus obtaining the call frequency of the block.

[0375] In step 2120, if it is determined that the number of blocks in the local cache node reaches the second condition, then based on the frequency of being called, the blocks to be cleared are identified from multiple blocks, and the blocks to be cleared are cleared in the local cache cluster.

[0376] In this specific implementation, firstly, for each local cache node in the local cache cluster, the number of cached blocks on the local cache node is counted to obtain the number of blocks on the local cache node. Next, the number of blocks is compared with the block threshold defined in the second condition. If the number of blocks is greater than or equal to the block threshold, it indicates that there are too many cached blocks on the local cache node, and the number of blocks in the local cache node has reached the second condition, requiring data clearing of the blocks on the local cache node. Based on this, according to the frequency of each block's invocation, a predetermined number of blocks with lower invocation frequency are selected as blocks to be cleared, and these blocks are cleared in the local cache cluster. The block threshold is used to limit the minimum number of blocks required to trigger data clearing, and the predetermined number indicates the number of blocks to be cleared in a single cache data eviction.

[0377] like Figure 22BThe diagram illustrates the process of evicting blocks from local cache node K based on their frequency of access. Specifically, local cache node K caches eight blocks: block 1 (accessed 131 times), block 21 (accessed 13 times), block 3 (accessed 22 times), block 14 (accessed 28 times), block 23 (accessed 80 times), block 6 (accessed 74 times), block 8 (accessed 18 times), and block 17 (accessed 43 times). Therefore, when evicting the three least accessed blocks from local cache node K, blocks 21, 3, and 8 are cleared.

[0378] The advantage of this embodiment is that it takes into account that if a certain block (cache data) was frequently accessed before the current time, then that block will likely be accessed frequently in the future. Therefore, based on the access frequency of each block, it distinguishes between blocks that were frequently accessed before the current time and blocks that were accessed less frequently before the current time. Thus, when the cache data eviction mechanism is triggered, the least accessed blocks are cleared. This approach can update the cache data of the local cache cluster in a timely manner and clear the less accessed cache data, thereby improving the cache space utilization efficiency of the local cache cluster.

[0379] In another embodiment, the specific process of updating the cached data of the local cache cluster may include, but is not limited to, the following steps:

[0380] For each block in each local cache node, determine the expiration time of each block;

[0381] If it is determined that the current time has expired, the block is cleared in the local cache cluster.

[0382] The expiration time refers to the specific point in time when a block should be removed from the local cache cluster.

[0383] In this specific implementation, the expiration time of each block is pre-set before being cached on the local cache node and is bound to each block. Based on this, with authorization, the block information of each block can be obtained from the background logs of the cache server in the local cache cluster, and the expiration time of the block can be extracted from the block information. The expiration time indicates the separate validity period. Next, for each block, the current time is checked in real-time to see if it has reached its separate expiration time. If it is determined that the current time has reached the expiration time, indicating that the validity period of the block has expired and it is no longer allowed to be accessed, the block is cleared from the local cache cluster. If the current time has not reached the expiration time, the block continues to be cached on the local cache node, so that the block can still be accessed at any time.

[0384] The advantage of this embodiment is that when each block is formed, an expiration time is set for each block, and the difference between the current time and the expiration time is detected in real time. When the current time reaches the expiration time, the block is cleared in the local cache cluster. This method can promptly clear expired blocks on the local cache nodes, thereby improving the rationality of block management and the space utilization efficiency of the local cache cluster.

[0385] The apparatus and device according to embodiments of this disclosure will now be described.

[0386] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0387] It should be noted that in various specific embodiments of this application, when processing is required based on data related to the characteristics of the target object, such as target object attribute information or a set of attribute information, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require obtaining target object attribute information, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of the embodiments of this application be obtained.

[0388] Figure 23 A schematic diagram of the structure of a cache data processing apparatus 2300 provided in an embodiment of this disclosure. The cache data processing apparatus 2300 includes:

[0389] The first acquisition unit 2310 is used to acquire the processing request of the target processing node for the target block in the target document, wherein the page in the target document is pre-divided into multiple blocks in the dimensions of rows and columns, the multiple blocks include the target block, and the processing request contains the block index of the target block.

[0390] The determining unit 2320 is used to determine the target local cache node corresponding to the block index in the local cache cluster bound to the target processing node, wherein the local cache cluster contains multiple local cache nodes, and the multiple local cache nodes contain the target local cache node.

[0391] The second acquisition unit 2330 is used to acquire the target block from the target local cache node if the target local cache node contains the target block;

[0392] The third acquisition unit 2340 is used to acquire the target block from the data source if the target local cache does not contain the target block.

[0393] Optionally, the cache data processing apparatus 2300 further includes a cache unit (not shown), which includes:

[0394] A partitioning module (not shown) is used to divide pages in the target document into multiple blocks along the dimensions of rows and columns;

[0395] A determination module (not shown) is used to determine the chunk index of a chunk based on the chunk's digest value;

[0396] A storage module (not shown) is used to store blocks into the local cache node corresponding to the block index.

[0397] Optionally, the determining module (not shown) is used for:

[0398] The remainder of the digest value modulo a first number is used as the block index, where the first number is the number of local cache nodes in the local cache cluster;

[0399] The storage module (not shown) is used for:

[0400] The blocks are stored in local cache nodes with the same sequence number as the block index.

[0401] Optionally, the determining module (not shown) is used for:

[0402] The summary value is used as the block index;

[0403] Local cache nodes have block index ranges, and the storage module (not shown) is used for:

[0404] The blocks are stored in the local cache node corresponding to the block index range where the block index is located.

[0405] Optionally, the block index range of the local cache node is determined in the following way:

[0406] Obtain the preset summary data body;

[0407] Determine the total number of cache nodes in the local cache cluster;

[0408] Based on the total number of cache nodes, the digest data body is divided into multiple digest data segments.

[0409] Multiple local cache nodes are matched with multiple digest data segments so that each local cache node corresponds to a digest data segment, and the digest data segment is used as the block index range corresponding to the local cache node.

[0410] Optionally, the determining module (not shown) is used for:

[0411] The summary value is used as the block index;

[0412] The storage module (not shown) is used for:

[0413] Determine the virtual digest node corresponding to each local cache node, wherein each virtual digest node has node digest information and each virtual digest node is located on the same preset digest data body;

[0414] For each virtual summary node, the difference between the block index and the node summary information is calculated to obtain the summary difference.

[0415] Based on the digest difference, the local cache node corresponding to the block index is determined, and the block is stored in the local cache node corresponding to the block index.

[0416] Optionally, the cached data processing apparatus 2300 further includes an exit unit (not shown), which is used for:

[0417] If the target local cache node is detected to have exited the local cache cluster, the virtual digest node corresponding to the target local cache node is cleared from the digest data body, and the digest value of each block in the target local cache node is determined.

[0418] Based on the summary value, the blocks are located on the summary data body to obtain the data position of the blocks;

[0419] Based on the data location, the blocks are re-stored on the local cache node corresponding to the virtual digest node whose distance from the data location meets the first condition.

[0420] Optionally, the cache data processing apparatus 2300 further includes a new unit (not shown), which is used for:

[0421] If a new cache node is detected in the local cache cluster, the new summary node corresponding to the new cache node is determined in the summary data body;

[0422] Determine the node distance between the newly added summary node and each virtual summary node;

[0423] Based on node distance, the target digest node is determined among multiple virtual digest nodes, and the local cache node corresponding to the target digest node is determined as the baseline cache node;

[0424] Multiple blocks in the baseline cache node are redistributed between the new cache node and the baseline cache node.

[0425] Optionally, multiple blocks in the baseline cache node may be reallocated between the new cache node and the baseline cache node, including:

[0426] Determine the first cache size of the newly added cache node, the second cache size of the baseline cache node, and the number of blocks in the partition;

[0427] The first ratio is determined based on the first cache size and the second cache size;

[0428] Based on the first ratio and the number of blocks, the blocks are redistributed among the new cache nodes and the baseline cache nodes.

[0429] Optionally, the cached data processing apparatus 2300 further includes a fault unit (not shown), which is used for:

[0430] If a node failure is detected in the local cache cluster, the virtual digest node corresponding to the failed cache node is located in the digest data body to obtain the target node location of the virtual digest node corresponding to the failed cache node.

[0431] Based on the target node location and the node location of the virtual digest node corresponding to each local cache node, the first cache node is determined among multiple local cache nodes;

[0432] The blocks in the faulty cache node are redistributed among multiple first cache nodes.

[0433] Optionally, the blocks in the failed cache node are redistributed among multiple first cache nodes, including:

[0434] Determine the cache node status of each first cache node and the block information of the blocks in the failed cache node;

[0435] Based on the cache node status and block information, a second cache node is selected from multiple first cache nodes, and the blocks from the faulty cache node are re-stored into the second cache node.

[0436] Optionally, the cached data processing device 2300 further includes a first update unit (not shown), which is used for:

[0437] For each block in each local cache node, determine the timestamp when the block was invoked;

[0438] If the number of blocks in the local cache node is determined to meet the second condition, the block to be cleared is identified from multiple blocks based on the timestamp of the call, and the block to be cleared is cleared in the local cache cluster.

[0439] Optionally, the cached data processing apparatus 2300 further includes a second update unit (not shown), which is used for:

[0440] For each block in each local cache node, determine the frequency of the block being called;

[0441] If the number of blocks in the local cache node is determined to meet the second condition, then based on the frequency of being called, the blocks to be cleared are identified from among the multiple blocks, and the blocks to be cleared are cleared in the local cache cluster.

[0442] Optionally, the cached data processing device 2300 further includes a third update unit (not shown), which is used for:

[0443] For each block in each local cache node, determine the expiration time of each block;

[0444] If it is determined that the current time has expired, the block is cleared in the local cache cluster.

[0445] Optionally, the cached data processing apparatus 2300 further includes a pre-storage unit (not shown), which is used for:

[0446] Determine the data retrieval frequency and data size for multiple reference data sets;

[0447] Based on the frequency of data access and the size of the data, alternative data is determined from multiple reference data, and the alternative data is cached in the local cache cluster.

[0448] Optionally, the processing request for the target chunk is generated by the target processing node based on the received processing task for the target document. The processing task for the target document is allocated to the target processing node by the task scheduling node in the target task cluster through the following process:

[0449] Obtain the current number of tasks and the current node status of each candidate processing node in the target task cluster;

[0450] For each candidate processing node, a first score is determined based on the current number of tasks;

[0451] Determine the second score based on the current node state;

[0452] Based on the first score and the second score, determine the total node score of the candidate processing node;

[0453] Based on the total node score, the target processing node is determined from multiple candidate processing nodes, and the processing task is assigned to the target processing node.

[0454] Optionally, the current node status of the candidate processing node includes the node latency and node resource status.

[0455] Based on the current node state, the second score is determined, including:

[0456] The first sub-score is determined based on the node latency.

[0457] The second sub-score is determined based on the node resource status;

[0458] The second score of the candidate processing node is determined based on the weighted sum of the first and second sub-scores.

[0459] Reference Figure 24 , Figure 24 To implement the terminal portion of the cached data processing method according to embodiments of this disclosure, the terminal includes: a radio frequency (RF) circuit 2410, a memory 2415, an input unit 2430, a display unit 2440, a sensor 2450, an audio circuit 2460, a wireless fidelity (WiFi) module 2470, a processor 2480, and a power supply 2490, among other components. Those skilled in the art will understand that... Figure 24 The terminal structure shown does not constitute a limitation on mobile phones or computers and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0460] The RF circuit 2410 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 2480; in addition, it transmits uplink data to the base station.

[0461] The memory 2415 can be used to store software programs and modules. The processor 2480 executes various functional applications and data processing of the target terminal by running the software programs and modules stored in the memory 2415.

[0462] The input unit 2430 can be used to receive input numeric or character information, and to generate key signal inputs related to the settings and function control of the target terminal. Specifically, the input unit 2430 may include a touch panel 2431 and other input devices 2432.

[0463] Display unit 2440 can be used to display input or provided information, as well as various menus of the target terminal. Display unit 2440 may include display panel 2441.

[0464] Audio circuitry 2460, speaker 2461, and microphone 2462 provide an audio interface.

[0465] In this embodiment, the processor 2480 included in the terminal can execute the cached data processing method of the previous embodiment.

[0466] The terminals disclosed in this embodiment include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The embodiments of this invention can be applied to various scenarios, including but not limited to data security, blockchain, data storage, and information technology.

[0467] Figure 25 This is a partial structural block diagram of a server for implementing the cached data processing method of this disclosure. The server can vary significantly due to different configurations or performance, and may include one or more Central Processing Units (CPUs) 2522 (e.g., one or more processors) and memory 2532, and one or more storage media 2530 (e.g., one or more mass storage devices) for storing application programs 2542 or data 2544. The memory 2532 and storage media 2530 can be temporary or persistent storage. The program stored in the storage media 2530 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 2522 may be configured to communicate with the storage media 2530 and execute the series of instruction operations in the storage media 2530 on the server.

[0468] The server may also include one or more power supplies 2523, one or more wired or wireless network interfaces 2550, one or more input / output interfaces 2558, and / or one or more operating systems 2541, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0469] The central processing unit 2522 in the server can be used to execute the cached data processing method of the present disclosure embodiments.

[0470] This disclosure also provides a computer-readable storage medium for storing program code for executing the cached data processing methods of the foregoing embodiments.

[0471] This disclosure also provides a computer program product comprising a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the cached data processing method described above.

[0472] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0473] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0474] It should be understood that in the description of the embodiments of this disclosure, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0475] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0476] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0477] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0478] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0479] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0480] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0481] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A method of processing data in a cache, the method comprising: The method includes: Obtain the processing request of the target processing node for the target block in the target document, wherein the page in the target document is pre-divided into multiple blocks in the dimensions of rows and columns, the multiple blocks include the target block, and the processing request contains the block index of the target block; In the local cache cluster bound to the target processing node, the target local cache node corresponding to the block index is determined, wherein the local cache cluster contains multiple local cache nodes, and the multiple local cache nodes include the target local cache node; If the target local cache node contains the target block, obtain the target block from the target local cache node; If the target local cache does not contain the target block, obtain the target block from the data source.

2. The cached data processing method according to claim 1, characterized in that, Before obtaining the processing request from the target processing node for the target chunk in the target document, the method further includes: The target document's pages are divided into the multiple blocks along the dimensions of rows and columns; Based on the digest value of the block, determine the block index; The blocks are stored in the local cache node corresponding to the block index.

3. The cached data processing method according to claim 2, characterized in that, Determining the block index based on the digest value of the block includes: determining the remainder of the digest value modulo a first number as the block index, wherein the first number is the number of local cache nodes in the local cache cluster; The step of storing the block in the local cache node corresponding to the block index includes: storing the block in the local cache node with the same sequence number as the block index.

4. The cached data processing method according to claim 2, characterized in that, Determining the block index based on the block's digest value includes: determining the digest value as the block index; The local cache node has a block index range, and storing the block in the local cache node corresponding to the block index includes: storing the block in the local cache node corresponding to the block index range where the block index is located.

5. The cached data processing method according to claim 4, characterized in that, The block index range of the local cache node is determined in the following way: Obtain the preset summary data body; Determine the total number of cache nodes in the local cache cluster; Based on the total number of cache nodes, the digest data body is divided into multiple digest data segments. Multiple local cache nodes are matched with multiple digest data segments so that each local cache node corresponds to one digest data segment, and the digest data segment is used as the block index range corresponding to the local cache node.

6. The cached data processing method according to claim 2, characterized in that, Determining the block index based on the block's digest value includes: determining the digest value as the block index; The step of storing the block in the local cache node corresponding to the block index includes: Determine the virtual digest node corresponding to each of the local cache nodes, wherein each of the virtual digest nodes has node digest information and each of the virtual digest nodes is located on the same preset digest data body; For each virtual summary node, the difference between the block index and the node summary information is calculated to obtain the summary difference. Based on the digest difference, the local cache node corresponding to the block index is determined, and the block is stored in the local cache node corresponding to the block index.

7. The cached data processing method according to claim 6, characterized in that, After storing the block in the local cache node corresponding to the block index, the method further includes: If the target local cache node is detected to have exited the local cache cluster, the virtual digest node corresponding to the target local cache node is cleared from the digest data body, and the digest value of each block in the target local cache node is determined. Based on the summary value, the block is located on the summary data body to obtain the data position of the block; Based on the data location, the block is re-stored on the local cache node corresponding to the virtual digest node whose distance from the data location meets the first condition.

8. The cached data processing method according to claim 6, characterized in that, After storing the block in the local cache node corresponding to the block index, the method further includes: If a new cache node is detected in the local cache cluster, the new summary node corresponding to the new cache node is determined in the summary data body; Determine the node distance between the newly added summary node and each of the virtual summary nodes; Based on the node distance, a target digest node is determined among the multiple virtual digest nodes, and the local cache node corresponding to the target digest node is determined as the base cache node; The multiple blocks in the baseline cache node are redistributed among the new cache node and the baseline cache node.

9. The cached data processing method according to claim 6, characterized in that, After storing the block in the local cache node corresponding to the block index, the method further includes: If a node failure is detected in the local cache cluster, the virtual digest node corresponding to the failed cache node is located on the digest data body to obtain the target node location of the virtual digest node corresponding to the failed cache node. Based on the target node location and the node location of the virtual digest node corresponding to each local cache node, the first cache node is determined among the multiple local cache nodes; The blocks in the faulty cache node are redistributed among multiple first cache nodes.

10. The cached data processing method according to claim 9, characterized in that, The step of redistributing the blocks in the faulty cache node among multiple first cache nodes includes: Determine the cache node status of each of the first cache nodes and the block information of the blocks in the faulty cache nodes; Based on the cache node status and the block information, a second cache node is selected from multiple first cache nodes, and the blocks in the faulty cache node are re-stored in the second cache node.

11. The cached data processing method according to claim 1, characterized in that, The blocks in each local cache node of the local cache cluster are updated in the following way: For each block in each local cache node, determine the timestamp of the block being invoked; If it is determined that the number of blocks in the local cache node reaches the second condition, then based on the invoked timestamp, the block to be cleared is determined from among the multiple blocks, and the block to be cleared is cleared in the local cache cluster.

12. The cached data processing method according to claim 1, characterized in that, The blocks in each local cache node of the local cache cluster are updated in the following way: For each block in each local cache node, determine the frequency of invocation of the block; If it is determined that the number of blocks in the local cache node reaches the second condition, then based on the frequency of being called, a block to be cleared is determined from among the multiple blocks, and the block to be cleared is cleared in the local cache cluster.

13. The cached data processing method according to claim 1, characterized in that, The blocks in each local cache node of the local cache cluster are updated in the following way: For each block in each local cache node, determine the expiration time of each block; If it is determined that the current time has reached the expiration time, then the block is cleared in the local cache cluster.

14. The cached data processing method according to claim 1, characterized in that, Before obtaining the processing request from the target processing node for the target chunk in the target document, the cached data processing method further includes: Determine the data retrieval frequency and data size for multiple reference data sets; Based on the data call frequency and the data size, candidate data is determined from the plurality of reference data, and the candidate data is cached in the local cache cluster.

15. A cached data processing apparatus, characterized in that, The device includes: The first acquisition unit is used to acquire the processing request of the target processing node for the target block in the target document, wherein the page in the target document is pre-divided into multiple blocks in the dimensions of rows and columns, the multiple blocks include the target block, and the processing request contains the block index of the target block; The determining unit is configured to determine the target local cache node corresponding to the block index in a local cache cluster bound to the target processing node, wherein the local cache cluster includes multiple local cache nodes, and the multiple local cache nodes include the target local cache node; The second acquisition unit is configured to acquire the target block from the target local cache node if the target local cache node contains the target block; The third acquisition unit is used to acquire the target block from the data source if the target local cache does not contain the target block.

16. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the cached data processing method according to any one of claims 1 to 14.

17. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the cached data processing method according to any one of claims 1 to 14.

18. A computer program product comprising a computer program that is read and executed by a processor of an electronic device, causing the electronic device to perform the cached data processing method according to any one of claims 1 to 14.

Images